[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:30:55 dist-test-slave-38pg krb5kdc[24629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:30:55 dist-test-slave-38pg krb5kdc[24629](info): set up 2 sockets
Sep 28 06:30:55 dist-test-slave-38pg krb5kdc[24629](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:30:56.418694 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:30:56 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:30:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:30:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:30:56.548228 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38601
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:56.639097 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:56.639381 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:56.639429 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:56.640795 24642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:30:56.640870 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:56.640895 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:56.640920 24642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:30:56.640952 24642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:30:56.642674 24642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:38601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:56.642992 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:56.643273 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:56.653087 24651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:56.653062 24649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:56.653086 24642 server_base.cc:1034] running on GCE node
W20260928 06:30:56.653066 24648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:56.653687 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:56.654050 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:56.655201 24642 hybrid_clock.cc:648] HybridClock initialized: now 1790577056655195 us; error 35 us; skew 500 ppm
Sep 28 06:30:56 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577056, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:56.657820 24642 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:30:56.658299 24642 webserver.cc:460] Webserver started at http://127.24.11.190:39573/ using document root <none> and password file <none>
I20260928 06:30:56.658560 24642 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:56.658631 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:56.658780 24642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:56.659888 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "c4f19a582f29474da9f5f2fa6d2e892d"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "d33ad9d7b049ee82e19ac9f84a12452f"
server_key_iv: "26fd32cd14328180fa11bc45e32071c7"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.661185 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "c4f19a582f29474da9f5f2fa6d2e892d"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "d33ad9d7b049ee82e19ac9f84a12452f"
server_key_iv: "26fd32cd14328180fa11bc45e32071c7"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.662753 24642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:30:56.663580 24658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:56.663759 24642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:30:56.663867 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "c4f19a582f29474da9f5f2fa6d2e892d"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "d33ad9d7b049ee82e19ac9f84a12452f"
server_key_iv: "26fd32cd14328180fa11bc45e32071c7"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.664018 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:56.678550 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:56.681092 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:56.681270 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:56.685206 24642 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:38601
I20260928 06:30:56.685247 24710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:38601 every 8 connection(s)
I20260928 06:30:56.685710 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:30:56.686640 24711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:56.689890 24711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: Bootstrap starting.
I20260928 06:30:56.690524 24711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:56.690821 24711 log.cc:824] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:56.691550 24711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: No bootstrap required, opened a new log
I20260928 06:30:56.693210 24711 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } }
I20260928 06:30:56.693383 24711 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:56.693425 24711 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f19a582f29474da9f5f2fa6d2e892d, State: Initialized, Role: FOLLOWER
I20260928 06:30:56.693665 24711 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } }
I20260928 06:30:56.693701 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 24642
I20260928 06:30:56.693774 24711 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:30:56.693807 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:30:56.693821 24711 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:30:56.693877 24711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:56.694406 24711 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } }
I20260928 06:30:56.694434 24622 external_mini_cluster.cc:1428] Setting key f910f3fd9a63c4a8cbb0e3d260386f05
I20260928 06:30:56.694525 24711 leader_election.cc:302] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f19a582f29474da9f5f2fa6d2e892d; no voters:
I20260928 06:30:56.694731 24711 leader_election.cc:288] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:30:56.694770 24714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:30:56.694900 24714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [term 1 LEADER]: Becoming Leader. State: Replica: c4f19a582f29474da9f5f2fa6d2e892d, State: Running, Role: LEADER
I20260928 06:30:56.695019 24711 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:30:56.694983 24714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } }
I20260928 06:30:56.695322 24716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f19a582f29474da9f5f2fa6d2e892d. Latest consensus state: current_term: 1 leader_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } } }
I20260928 06:30:56.695569 24716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [sys.catalog]: This master's current role is: LEADER
I20260928 06:30:56.695504 24715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f19a582f29474da9f5f2fa6d2e892d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38601 } } }
I20260928 06:30:56.695824 24715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d [sys.catalog]: This master's current role is: LEADER
W20260928 06:30:56.697150 24731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:30:56.697216 24731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:30:56.697417 24719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:30:56.697607 24719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:30:56.699280 24719 catalog_manager.cc:1347] Generated new cluster ID: f84e7815b0634449bcfb8228920afe58
I20260928 06:30:56.699335 24719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:30:56.706184 24719 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:30:56.706881 24719 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 06:30:56 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:56.712656 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:56.695689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.24.11.190:38601)
0928 06:30:56.695868 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:56.695878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:56.695921 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:30:56.698193 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:56.698199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:56.698242 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:56.698405 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:56.700252 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.702278 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.704273 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.704735 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.704865 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:56.708868 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:56.708872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:56.708873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:56.708893 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:56.710708 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:56.711182 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:56.711185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:56.711185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:56.711287 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:56.711587 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:56.711587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:56.711588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:56.711811 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:56.711983 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:56.712269 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:30:56.712427 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260928 06:30:56.719451 24719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f19a582f29474da9f5f2fa6d2e892d: Generated new TSK 0
I20260928 06:30:56.719647 24719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:30:56.749400 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:56.838819 24737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:56.839085 24737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:56.839118 24737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:56.840517 24737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:56.840584 24737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:56.840626 24737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:30:56.842500 24737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:56.842816 24737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:56.843135 24737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:56.849659 24742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:56.849738 24737 server_base.cc:1034] running on GCE node
W20260928 06:30:56.849679 24743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:56.849691 24745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:56.850173 24737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:56.850476 24737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:56.851668 24737 hybrid_clock.cc:648] HybridClock initialized: now 1790577056851656 us; error 43 us; skew 500 ppm
Sep 28 06:30:56 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577056, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:56.854662 24737 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:30:56.855182 24737 webserver.cc:460] Webserver started at http://127.24.11.129:39943/ using document root <none> and password file <none>
I20260928 06:30:56.855436 24737 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:56.855507 24737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:56.855664 24737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:56.856786 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "1f51e440e7c4414d90d67d792f9d7948"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "8744157ac32cdf663c39ddf414d8c549"
server_key_iv: "5399e87c77ac86a44275dc8bb540e2a2"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.857167 24737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "1f51e440e7c4414d90d67d792f9d7948"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "8744157ac32cdf663c39ddf414d8c549"
server_key_iv: "5399e87c77ac86a44275dc8bb540e2a2"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.858698 24737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:30:56.859524 24752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:56.859721 24737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:30:56.859834 24737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "1f51e440e7c4414d90d67d792f9d7948"
format_stamp: "Formatted at 2026-09-28 06:30:56 on dist-test-slave-38pg"
server_key: "8744157ac32cdf663c39ddf414d8c549"
server_key_iv: "5399e87c77ac86a44275dc8bb540e2a2"
server_key_version: "encryptionkey@0"
I20260928 06:30:56.859916 24737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:56.887761 24737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:56.890455 24737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:56.890656 24737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:56.890941 24737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:56.891263 24737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:56.891320 24737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:56.891363 24737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:56.891388 24737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:56.897742 24737 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:43027
I20260928 06:30:56.897794 24866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:43027 every 8 connection(s)
I20260928 06:30:56.898175 24737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:30:56 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577056, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:56.905224 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 24737
I20260928 06:30:56.905350 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:30:56.905567 24622 external_mini_cluster.cc:1428] Setting key ad6e3f50e906f54c1613f7de3ef2ef63
I20260928 06:30:56.908355 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:56.898977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:50945 (local address 127.24.11.190:38601)
0928 06:30:56.899052 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:56.899070 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:56.899276 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:30:56.899797 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:56.899798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:56.899810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:56.899858 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:56.900879 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.901387 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.902516 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.902740 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.902827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:56.904963 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:56.904965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:56.904965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:56.904972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:56.906893 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:56.907305 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:56.907307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:56.907307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:56.907377 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:56.907690 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:56.907691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:56.907691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:56.907804 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:56.907862 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:56.908217 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:30:56.908290 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:30:56.908666 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:56.898952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38601 (local address 127.24.11.129:50945)
0928 06:30:56.899140 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:56.899182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:56.899686 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:56.899947 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:56.899957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:56.900545 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:56.900737 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:56.900742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.901488 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.901489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:56.902416 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:56.902418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:56.902843 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:56.902844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:56.902967 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:56.903444 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:56.903453 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:56.904872 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:56.906982 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:56.906987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:56.906989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:56.907216 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:56.907450 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:56.907451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:56.907451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:56.907610 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:56.907873 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:56.907879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:56.908069 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 06:30:56.908195 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:30:56.908388 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 06:30:56.909240 24867 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38601
I20260928 06:30:56.909337 24867 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:56.909608 24867 heartbeater.cc:511] Master 127.24.11.190:38601 requested a full tablet report, sending...
I20260928 06:30:56.910136 24675 ts_manager.cc:194] Registered new tserver with Master: 1f51e440e7c4414d90d67d792f9d7948 (127.24.11.129:43027)
I20260928 06:30:56.910743 24675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:50945
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:30:56.930677 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:57.031450 24872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:57.031664 24872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:57.031692 24872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:57.032810 24872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:57.032850 24872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:57.032876 24872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:30:57.034502 24872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:57.034812 24872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:57.035064 24872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:57.040199 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:57.040211 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:57.040287 24872 server_base.cc:1034] running on GCE node
W20260928 06:30:57.040210 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:57.040609 24872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:57.040853 24872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:57.042049 24872 hybrid_clock.cc:648] HybridClock initialized: now 1790577057042036 us; error 48 us; skew 500 ppm
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:57.044646 24872 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:30:57.045137 24872 webserver.cc:460] Webserver started at http://127.24.11.130:42135/ using document root <none> and password file <none>
I20260928 06:30:57.045357 24872 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:57.045403 24872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:57.045528 24872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:57.046648 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "20e891ed9fd54f878556cca9f7ba9dcc"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "b3662d31cb58bfb20dac825a99b0ea17"
server_key_iv: "0f7ee413dfa42ac568e87ee4d738149a"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.047101 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "20e891ed9fd54f878556cca9f7ba9dcc"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "b3662d31cb58bfb20dac825a99b0ea17"
server_key_iv: "0f7ee413dfa42ac568e87ee4d738149a"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.048463 24872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:30:57.049223 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.049397 24872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:30:57.049499 24872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "20e891ed9fd54f878556cca9f7ba9dcc"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "b3662d31cb58bfb20dac825a99b0ea17"
server_key_iv: "0f7ee413dfa42ac568e87ee4d738149a"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.049612 24872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:57.056828 24872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:57.059001 24872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:57.059163 24872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:57.059415 24872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:57.059731 24872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:57.059799 24872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.059837 24872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:57.059862 24872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.067091 24872 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:40507
I20260928 06:30:57.067126 25001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:40507 every 8 connection(s)
I20260928 06:30:57.067543 24872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:57.076206 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 24872
I20260928 06:30:57.076345 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:30:57.076527 24622 external_mini_cluster.cc:1428] Setting key 994c071be17295982786a870b39ac03d
I20260928 06:30:57.077112 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.068342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35317 (local address 127.24.11.190:38601)
0928 06:30:57.068410 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:57.068412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:57.068605 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:30:57.069074 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:57.069075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:57.069084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:57.069141 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:57.070091 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.070573 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.071583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.071768 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.071821 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.074032 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:57.074034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:57.074035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:57.074043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:57.075547 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.076001 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.076002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.076003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.076084 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.076362 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.076363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.076363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.076507 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:57.076587 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:57.076930 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 06:30:57.077022 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:30:57.077310 25004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.068325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38601 (local address 127.24.11.130:35317)
0928 06:30:57.068477 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:57.068518 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:57.068991 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:57.069197 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:57.069200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:57.069801 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:57.070004 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:57.070009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.070663 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.070664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:57.071495 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:57.071497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.071839 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.071840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:57.071968 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.072443 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:57.072451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:57.073912 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:57.075658 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:57.075667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:57.075669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:57.075909 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:57.076158 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:57.076159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:57.076160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:57.076280 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:57.076576 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:57.076583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:57.076779 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 06:30:57.076896 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:30:57.077076 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 06:30:57.077847 25002 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38601
I20260928 06:30:57.077963 25002 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:57.078163 25002 heartbeater.cc:511] Master 127.24.11.190:38601 requested a full tablet report, sending...
I20260928 06:30:57.078569 24675 ts_manager.cc:194] Registered new tserver with Master: 20e891ed9fd54f878556cca9f7ba9dcc (127.24.11.130:40507)
I20260928 06:30:57.079089 24675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:35317
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:30:57.104729 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:57.209486 25007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:57.209900 25007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:57.209931 25007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:57.211194 25007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:57.211242 25007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:57.211272 25007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:30:57.212990 25007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:57.213233 25007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:57.213445 25007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:57.219352 25012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:57.219442 25007 server_base.cc:1034] running on GCE node
W20260928 06:30:57.219621 25013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:57.219352 25015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:57.219902 25007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:57.220175 25007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:57.221336 25007 hybrid_clock.cc:648] HybridClock initialized: now 1790577057221323 us; error 29 us; skew 500 ppm
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:57.224100 25007 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:30:57.224633 25007 webserver.cc:460] Webserver started at http://127.24.11.131:42025/ using document root <none> and password file <none>
I20260928 06:30:57.224931 25007 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:57.225030 25007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:57.225184 25007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:57.226411 25007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "4f38d9bbdc3a44f99c6fc8255db8cdc7"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "0f7efec882a0118856368f1c2bd07188"
server_key_iv: "f7d2ac17bf64ac7ffa41d025443a6906"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.226938 25007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "4f38d9bbdc3a44f99c6fc8255db8cdc7"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "0f7efec882a0118856368f1c2bd07188"
server_key_iv: "f7d2ac17bf64ac7ffa41d025443a6906"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.228372 25007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:30:57.229132 25022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.229305 25007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:30:57.229415 25007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "4f38d9bbdc3a44f99c6fc8255db8cdc7"
format_stamp: "Formatted at 2026-09-28 06:30:57 on dist-test-slave-38pg"
server_key: "0f7efec882a0118856368f1c2bd07188"
server_key_iv: "f7d2ac17bf64ac7ffa41d025443a6906"
server_key_version: "encryptionkey@0"
I20260928 06:30:57.229496 25007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:57.243091 25007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:57.244989 25007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:57.245164 25007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:57.245456 25007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:57.245836 25007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:57.245872 25007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.245893 25007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:57.245906 25007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:57.260360 25007 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37133
I20260928 06:30:57.260397 25136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37133 every 8 connection(s)
I20260928 06:30:57.260771 25007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:57.270399 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25007
I20260928 06:30:57.270496 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:30:57.270566 24720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.261630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:57933 (local address 127.24.11.190:38601)
0928 06:30:57.261687 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:57.261689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:57.262044 (+ 355us) server_negotiation.cc:407] Connection header received
0928 06:30:57.262630 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:57.262632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:57.262645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:57.262702 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:57.263769 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.264278 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.265285 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.265500 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.265614 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.267852 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:57.267853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:57.267853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:57.267860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:57.269084 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.269444 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.269445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.269445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.269496 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.269873 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.269874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.269875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.270011 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:57.270079 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:57.270367 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:30:57.270455 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:30:57.270668 24622 external_mini_cluster.cc:1428] Setting key 2554d4e2a88a3ba27c1ca53601fa5ba2
I20260928 06:30:57.270689 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.261631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38601 (local address 127.24.11.131:57933)
0928 06:30:57.261780 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:57.261816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:57.262526 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:57.262778 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:57.262782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:57.263397 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:57.263643 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:57.263648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.264376 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.264378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:57.265189 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:57.265191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.265624 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.265625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:57.265746 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.266285 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:57.266294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:57.267724 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:57.269155 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:57.269160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:57.269162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:57.269376 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:57.269655 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:57.269655 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:57.269656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:57.269781 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:57.270078 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:57.270082 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:57.270211 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:30:57.270294 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:30:57.270414 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 06:30:57.271771 25137 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38601
I20260928 06:30:57.271879 25137 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:57.272064 25137 heartbeater.cc:511] Master 127.24.11.190:38601 requested a full tablet report, sending...
I20260928 06:30:57.272439 24675 ts_manager.cc:194] Registered new tserver with Master: 4f38d9bbdc3a44f99c6fc8255db8cdc7 (127.24.11.131:37133)
I20260928 06:30:57.272814 24675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:57933
I20260928 06:30:57.283015 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:30:57.309926 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.302011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.24.11.129:43027)
0928 06:30:57.302181 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:57.302186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:57.302444 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:30:57.302480 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:57.302482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:57.302496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:57.302742 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:57.303321 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.304053 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.304784 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.305111 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.305176 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.307423 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:57.307434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:57.307435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:57.307442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:57.308906 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.309215 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.309218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.309218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.309270 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.309436 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.309437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.309437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.309617 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:57.309745 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:57.309782 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:30:57.309843 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
W20260928 06:30:57.310527 24780 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44358
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[24629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:30:57.335949 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:57.328163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.24.11.129:43027)
0928 06:30:57.328254 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:57.328256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:57.328358 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:30:57.328456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:57.328457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:57.328477 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:57.328542 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:57.329108 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.329699 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.330452 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:57.330661 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:57.330745 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:57.333004 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:57.333005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:57.333005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:57.333014 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:57.334621 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.335114 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.335115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.335116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.335190 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:57.335399 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:57.335400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:57.335400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:57.335513 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:57.335643 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:57.335743 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:30:57.335839 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:30:57.336712 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 24737
I20260928 06:30:57.341519 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 24872
I20260928 06:30:57.346982 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25007
I20260928 06:30:57.351400 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 24642
2026-09-28T06:30:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1999 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[25151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[25151](info): set up 2 sockets
Sep 28 06:30:57 dist-test-slave-38pg krb5kdc[25151](info): commencing operation
krb5kdc: starting...
W20260928 06:30:58.390312 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:30:58.512038 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41149
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:58.593003 25164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:58.593231 25164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:58.593254 25164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:58.594401 25164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:30:58.594441 25164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:58.594455 25164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:58.594468 25164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:30:58.594480 25164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:30:58.595902 25164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:58.596131 25164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:58.596381 25164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:58.601403 25170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:58.601505 25164 server_base.cc:1034] running on GCE node
W20260928 06:30:58.601408 25169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:58.601425 25172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:58.601976 25164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:58.602242 25164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:58.603387 25164 hybrid_clock.cc:648] HybridClock initialized: now 1790577058603376 us; error 29 us; skew 500 ppm
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:58.606031 25164 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:30:58.606514 25164 webserver.cc:460] Webserver started at http://127.24.11.190:34917/ using document root <none> and password file <none>
I20260928 06:30:58.606747 25164 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:58.606794 25164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:58.606966 25164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:58.608062 25164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "067438b95bbf41748ab89145242eafa1"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "383552888fb6fab29c8fe86beaf3b97c"
server_key_iv: "a499f9d332a7e9940dd3b25fe5ce04c8"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.608434 25164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "067438b95bbf41748ab89145242eafa1"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "383552888fb6fab29c8fe86beaf3b97c"
server_key_iv: "a499f9d332a7e9940dd3b25fe5ce04c8"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.609839 25164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:30:58.610605 25179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:58.610805 25164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:30:58.610920 25164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "067438b95bbf41748ab89145242eafa1"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "383552888fb6fab29c8fe86beaf3b97c"
server_key_iv: "a499f9d332a7e9940dd3b25fe5ce04c8"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.610997 25164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:58.631811 25164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:58.633948 25164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:58.634120 25164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:58.638336 25164 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:41149
I20260928 06:30:58.638407 25232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:41149 every 8 connection(s)
I20260928 06:30:58.638760 25164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:30:58.639627 25233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:58.641842 25233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1: Bootstrap starting.
I20260928 06:30:58.642405 25233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:58.642629 25233 log.cc:824] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:58.643356 25233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1: No bootstrap required, opened a new log
I20260928 06:30:58.644876 25233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } }
I20260928 06:30:58.645012 25233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:58.645035 25233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067438b95bbf41748ab89145242eafa1, State: Initialized, Role: FOLLOWER
I20260928 06:30:58.645176 25233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } }
I20260928 06:30:58.645258 25233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:30:58.645289 25233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:30:58.645324 25233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:58.645902 25233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } }
I20260928 06:30:58.645988 25233 leader_election.cc:302] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 067438b95bbf41748ab89145242eafa1; no voters:
I20260928 06:30:58.646121 25233 leader_election.cc:288] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:30:58.646169 25236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:30:58.646339 25236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [term 1 LEADER]: Becoming Leader. State: Replica: 067438b95bbf41748ab89145242eafa1, State: Running, Role: LEADER
I20260928 06:30:58.646405 25233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:30:58.646430 25236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } }
I20260928 06:30:58.646818 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 067438b95bbf41748ab89145242eafa1. Latest consensus state: current_term: 1 leader_uuid: "067438b95bbf41748ab89145242eafa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } } }
I20260928 06:30:58.646950 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:30:58.647078 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25164
I20260928 06:30:58.647171 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:30:58.647330 24622 external_mini_cluster.cc:1428] Setting key 121f78a2a59cd098b6a5c241c0d99356
I20260928 06:30:58.647595 25245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:30:58.648176 25245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:30:58.647993 25237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "067438b95bbf41748ab89145242eafa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067438b95bbf41748ab89145242eafa1" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41149 } } }
I20260928 06:30:58.648258 25237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:30:58.649847 25245 catalog_manager.cc:1347] Generated new cluster ID: 7250958e7b004aa49b476ec23776f16c
I20260928 06:30:58.649901 25245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:58.658190 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:58.647989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.24.11.190:41149)
0928 06:30:58.648148 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:58.648159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:58.648195 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:30:58.648423 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:58.648429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:58.648482 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:58.648641 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:58.649352 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.650487 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.651235 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.651682 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.651801 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:58.654204 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:58.654208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:58.654209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:58.654228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:58.656492 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:58.656878 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:58.656881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:58.656882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:58.656987 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:58.657204 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:58.657205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:58.657205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:58.657459 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:58.657654 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:58.657768 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:30:58.658014 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260928 06:30:58.679354 25245 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:30:58.679978 25245 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:30:58.684724 25245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 067438b95bbf41748ab89145242eafa1: Generated new TSK 0
I20260928 06:30:58.684940 25245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:30:58.720651 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:58.811043 25259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:58.811265 25259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:58.811291 25259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:58.812500 25259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:58.812548 25259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:58.812577 25259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:30:58.814219 25259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:58.814474 25259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:58.814736 25259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:58.819849 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:58.819939 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:58.819933 25267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:58.820212 25259 server_base.cc:1034] running on GCE node
I20260928 06:30:58.820442 25259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:58.820688 25259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:58.821856 25259 hybrid_clock.cc:648] HybridClock initialized: now 1790577058821843 us; error 28 us; skew 500 ppm
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:58.824565 25259 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:30:58.825101 25259 webserver.cc:460] Webserver started at http://127.24.11.129:39221/ using document root <none> and password file <none>
I20260928 06:30:58.825340 25259 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:58.825409 25259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:58.825613 25259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:58.826727 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "4d61c298fadd913adc6757b5be9952ab"
server_key_iv: "d9b6a9eb28ef16a48b73a93bd61acb03"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.827133 25259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "4d61c298fadd913adc6757b5be9952ab"
server_key_iv: "d9b6a9eb28ef16a48b73a93bd61acb03"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.828553 25259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:30:58.829280 25274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:58.829566 25259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:30:58.829689 25259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "4d61c298fadd913adc6757b5be9952ab"
server_key_iv: "d9b6a9eb28ef16a48b73a93bd61acb03"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.829777 25259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:58.851082 25259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:58.853678 25259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:58.853861 25259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:58.854125 25259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:58.854719 25259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:58.854852 25259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:58.854905 25259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:58.854941 25259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:58.860805 25259 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39121
I20260928 06:30:58.860846 25387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39121 every 8 connection(s)
I20260928 06:30:58.861169 25259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:30:58.866452 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25259
I20260928 06:30:58.866557 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:30:58.866715 24622 external_mini_cluster.cc:1428] Setting key 674be8b2d0f7bb10f64d7d9f94b37881
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
I20260928 06:30:58.874586 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:58.861910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60039 (local address 127.24.11.190:41149)
0928 06:30:58.861981 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:58.861983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:58.862166 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:30:58.866148 (+ 3982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:58.866150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:58.866162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:58.866230 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:58.867191 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.867767 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.869195 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.869407 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.869495 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:58.871688 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:58.871689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:58.871690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:58.871697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:58.873139 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:58.873596 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:58.873598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:58.873598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:58.873661 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:58.873923 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:58.873923 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:58.873924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:58.874041 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:58.874113 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:58.874408 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 06:30:58.874494 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:30:58.874814 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:58.861867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41149 (local address 127.24.11.129:60039)
0928 06:30:58.862029 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:58.862073 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:58.865945 (+ 3872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:58.866346 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:58.866351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:58.866751 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:58.867100 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:58.867105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.867879 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.867880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:58.869104 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:58.869106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:58.869504 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:58.869505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:58.869652 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:58.870209 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:58.870219 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:58.871593 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:58.873242 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:58.873249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:58.873251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:58.873483 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:58.873729 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:58.873730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:58.873730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:58.873847 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:58.874115 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:58.874122 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:58.874271 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:30:58.874373 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:30:58.874550 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260928 06:30:58.875391 25388 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41149
I20260928 06:30:58.875520 25388 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:58.875705 25388 heartbeater.cc:511] Master 127.24.11.190:41149 requested a full tablet report, sending...
I20260928 06:30:58.876283 25197 ts_manager.cc:194] Registered new tserver with Master: 702c66cbc3394e6bb0ca5f0a7ebbde21 (127.24.11.129:39121)
I20260928 06:30:58.876863 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:60039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:30:58.893123 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:58.975553 25393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:58.975782 25393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:58.975807 25393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:58.977034 25393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:58.977082 25393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:58.977110 25393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:30:58.978747 25393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:58.979015 25393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:58.979285 25393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:30:58.984717 25399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:58.984710 25401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:58.984788 25393 server_base.cc:1034] running on GCE node
W20260928 06:30:58.984710 25398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:58.985191 25393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:58.985404 25393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:58.986589 25393 hybrid_clock.cc:648] HybridClock initialized: now 1790577058986575 us; error 32 us; skew 500 ppm
Sep 28 06:30:58 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:58.989264 25393 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:30:58.989809 25393 webserver.cc:460] Webserver started at http://127.24.11.130:44399/ using document root <none> and password file <none>
I20260928 06:30:58.990043 25393 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:58.990089 25393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:58.990235 25393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:58.991304 25393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "df47c31099c0474283ee52bcee875f60"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "40aa8b2b575bc562f201248d9d534109"
server_key_iv: "aa6546b4a07163521e68285521965bde"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.991688 25393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "df47c31099c0474283ee52bcee875f60"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "40aa8b2b575bc562f201248d9d534109"
server_key_iv: "aa6546b4a07163521e68285521965bde"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.993052 25393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:30:58.993919 25408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:58.994199 25393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:30:58.994274 25393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "df47c31099c0474283ee52bcee875f60"
format_stamp: "Formatted at 2026-09-28 06:30:58 on dist-test-slave-38pg"
server_key: "40aa8b2b575bc562f201248d9d534109"
server_key_iv: "aa6546b4a07163521e68285521965bde"
server_key_version: "encryptionkey@0"
I20260928 06:30:58.994370 25393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:59.014549 25393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:59.016705 25393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:59.016819 25393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:59.017098 25393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:59.017442 25393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:59.017499 25393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:59.017565 25393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:59.017611 25393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:59.023463 25393 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:40627
I20260928 06:30:59.023510 25521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:40627 every 8 connection(s)
I20260928 06:30:59.023854 25393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:30:59.028216 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25393
I20260928 06:30:59.028324 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:30:59.028527 24622 external_mini_cluster.cc:1428] Setting key 6a80a1017d71ef48d82b0ea7b7796b23
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:59.034282 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.024679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:38625 (local address 127.24.11.190:41149)
0928 06:30:59.024756 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.024758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.025002 (+ 244us) server_negotiation.cc:407] Connection header received
0928 06:30:59.026016 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.026017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.026028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.026102 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.026708 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.027213 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.028349 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.028602 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.028696 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.031015 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.031017 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.031017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.031027 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.032581 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.033096 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.033097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.033097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.033148 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.033474 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.033475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.033476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.033645 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.033714 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.034131 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.034213 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:30:59.034600 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.024658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41149 (local address 127.24.11.130:38625)
0928 06:30:59.024818 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.024861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.025892 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.026161 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.026164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.026428 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:59.026619 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.026623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.027295 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.027296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.028205 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.028208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.028690 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.028691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.028822 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.029437 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:59.029445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:59.030917 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:59.032665 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:59.032672 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:59.032675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:59.033020 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:59.033216 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:59.033217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:59.033217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:59.033383 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:59.033748 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:59.033756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:59.033943 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.034080 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.034283 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 06:30:59.035127 25522 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41149
I20260928 06:30:59.035220 25522 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:59.035427 25522 heartbeater.cc:511] Master 127.24.11.190:41149 requested a full tablet report, sending...
I20260928 06:30:59.035880 25197 ts_manager.cc:194] Registered new tserver with Master: df47c31099c0474283ee52bcee875f60 (127.24.11.130:40627)
I20260928 06:30:59.036273 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:38625
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:30:59.057315 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:30:59.139751 25527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:30:59.139984 25527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:30:59.140022 25527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:30:59.141193 25527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:30:59.141237 25527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:30:59.141273 25527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:30:59.142825 25527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41149
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:30:59.143020 25527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:30:59.143200 25527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:30:59.148319 25527 server_base.cc:1034] running on GCE node
W20260928 06:30:59.148257 25533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:59.148283 25532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:30:59.148257 25535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:30:59.148814 25527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:30:59.149027 25527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:30:59.150226 25527 hybrid_clock.cc:648] HybridClock initialized: now 1790577059150212 us; error 43 us; skew 500 ppm
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:30:59.152972 25527 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:30:59.153465 25527 webserver.cc:460] Webserver started at http://127.24.11.131:33443/ using document root <none> and password file <none>
I20260928 06:30:59.153736 25527 fs_manager.cc:359] Metadata directory not provided
I20260928 06:30:59.153792 25527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:30:59.153971 25527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:30:59.155066 25527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "124f23086b1548b286f1beb5835aa9c0"
format_stamp: "Formatted at 2026-09-28 06:30:59 on dist-test-slave-38pg"
server_key: "1bc116ae0c9523b3e3f535fb54b3d945"
server_key_iv: "ae3bb6c53157edf38d720fc03ab10ba9"
server_key_version: "encryptionkey@0"
I20260928 06:30:59.155442 25527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "124f23086b1548b286f1beb5835aa9c0"
format_stamp: "Formatted at 2026-09-28 06:30:59 on dist-test-slave-38pg"
server_key: "1bc116ae0c9523b3e3f535fb54b3d945"
server_key_iv: "ae3bb6c53157edf38d720fc03ab10ba9"
server_key_version: "encryptionkey@0"
I20260928 06:30:59.156818 25527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:30:59.157714 25542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:59.157946 25527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:30:59.158058 25527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "124f23086b1548b286f1beb5835aa9c0"
format_stamp: "Formatted at 2026-09-28 06:30:59 on dist-test-slave-38pg"
server_key: "1bc116ae0c9523b3e3f535fb54b3d945"
server_key_iv: "ae3bb6c53157edf38d720fc03ab10ba9"
server_key_version: "encryptionkey@0"
I20260928 06:30:59.158145 25527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:30:59.194869 25527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:30:59.196895 25527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:30:59.197224 25527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:30:59.197602 25527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:30:59.198045 25527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:30:59.198102 25527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:59.198143 25527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:30:59.198171 25527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:30:59.204458 25527 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:38299
I20260928 06:30:59.204504 25655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:38299 every 8 connection(s)
I20260928 06:30:59.204841 25527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:30:59.212682 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25527
I20260928 06:30:59.212802 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:30:59.212953 24622 external_mini_cluster.cc:1428] Setting key 31eb3c8426bf0999c9df1fd17e99f36f
I20260928 06:30:59.214912 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.205584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:44575 (local address 127.24.11.190:41149)
0928 06:30:59.205642 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.205644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.205881 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:30:59.206849 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.206851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.206863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.206913 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.207620 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.208173 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.209197 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.209415 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.209510 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.211842 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.211843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.211844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.211852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.213412 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.213985 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.213986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.213986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.214040 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.214245 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.214246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.214246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.214375 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.214424 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.214724 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.214837 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:30:59.215037 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.205535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41149 (local address 127.24.11.131:44575)
0928 06:30:59.205718 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.205755 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.206727 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.207003 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.207006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.207279 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:30:59.207521 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.207525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.208274 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.208275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.209104 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.209106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.209519 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.209521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.209673 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.210243 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:30:59.210252 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:30:59.211731 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:30:59.213485 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:59.213493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:59.213495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:59.213919 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:59.214084 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:30:59.214085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:30:59.214085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:30:59.214185 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:30:59.214430 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:30:59.214434 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:30:59.214561 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.214640 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.214816 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260928 06:30:59.215509 25656 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41149
I20260928 06:30:59.215612 25656 heartbeater.cc:464] Registering TS with master...
I20260928 06:30:59.215859 25656 heartbeater.cc:511] Master 127.24.11.190:41149 requested a full tablet report, sending...
I20260928 06:30:59.216239 25197 ts_manager.cc:194] Registered new tserver with Master: 124f23086b1548b286f1beb5835aa9c0 (127.24.11.131:38299)
I20260928 06:30:59.216578 25197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:44575
I20260928 06:30:59.224795 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:30:59.232834 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.226553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.24.11.190:41149)
0928 06:30:59.226618 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.226624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.226658 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:30:59.226730 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.226731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.226740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.226792 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.227338 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.227818 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.228648 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.228850 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.228914 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.230286 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.230288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.230288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.230295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.231785 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.232158 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.232158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.232159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.232211 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.232366 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.232366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.232367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.232509 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.232581 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.232671 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.232732 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:30:59.235229 25197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:30:59.237152 25197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:30:59.243955 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.24.11.130:40627)
0928 06:30:59.239990 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.239995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.240068 (+ 73us) server_negotiation.cc:407] Connection header received
0928 06:30:59.240230 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.240233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.240245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.240336 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:59.240536 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241715 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243007 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243519 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243614 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243677 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.243820 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.243864 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260928 06:30:59.243947 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40627 (local address 127.0.0.1:45726)
0928 06:30:59.239964 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.240007 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.240090 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.240349 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.240352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.240354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:59.240450 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.240453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241827 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.241828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.242919 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.242921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243613 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.243692 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243706 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.243801 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.243869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260928 06:30:59.244122 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:38299 (local address 127.0.0.1:51692)
0928 06:30:59.240063 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.240093 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.240152 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.240584 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.240585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.240587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:59.240621 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.240623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241864 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.241865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.242806 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.242809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243783 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.243848 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243852 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.243905 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.243946 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":90,"threads_started":1}
I20260928 06:30:59.244113 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.24.11.129:39121)
0928 06:30:59.240058 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.240062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.240324 (+ 262us) server_negotiation.cc:407] Connection header received
0928 06:30:59.240377 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.240379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.240393 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.240465 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:59.240756 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241427 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.242997 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243668 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243727 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243787 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.243990 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.244031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 06:30:59.244113 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.24.11.131:38299)
0928 06:30:59.239904 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.239909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.240360 (+ 451us) server_negotiation.cc:407] Connection header received
0928 06:30:59.240399 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.240401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.240417 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.240492 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:59.240694 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241755 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.242891 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243611 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243687 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243745 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.243990 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.244031 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260928 06:30:59.244302 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.239766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39121 (local address 127.0.0.1:56536)
0928 06:30:59.240252 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.240259 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.240305 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.240616 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.240617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.240619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:59.240677 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.240679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.241572 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.241573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.242888 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.242890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.243815 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.243815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.243865 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.243868 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.243913 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.243948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":67,"threads_started":1}
I20260928 06:30:59.244978 25590 tablet_service.cc:1467] Processing CreateTablet for tablet 41813d5bc4a5449686ec69cf31d3166a (DEFAULT_TABLE table=test-table [id=3ae4863ce7a74a32ba82b8864ddee1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:59.244980 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 41813d5bc4a5449686ec69cf31d3166a (DEFAULT_TABLE table=test-table [id=3ae4863ce7a74a32ba82b8864ddee1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:59.245352 25590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41813d5bc4a5449686ec69cf31d3166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:59.245352 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41813d5bc4a5449686ec69cf31d3166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:59.249105 25674 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: Bootstrap starting.
I20260928 06:30:59.249239 25673 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Bootstrap starting.
I20260928 06:30:59.249500 25456 tablet_service.cc:1467] Processing CreateTablet for tablet 41813d5bc4a5449686ec69cf31d3166a (DEFAULT_TABLE table=test-table [id=3ae4863ce7a74a32ba82b8864ddee1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:30:59.249931 25674 tablet_bootstrap.cc:654] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:59.249970 25456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41813d5bc4a5449686ec69cf31d3166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:30:59.249962 25673 tablet_bootstrap.cc:654] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:59.250433 25673 log.cc:824] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:59.250496 25674 log.cc:824] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:59.251194 25673 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: No bootstrap required, opened a new log
I20260928 06:30:59.251267 25674 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: No bootstrap required, opened a new log
I20260928 06:30:59.251281 25673 ts_tablet_manager.cc:1397] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:30:59.251339 25674 ts_tablet_manager.cc:1397] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:30:59.252909 25673 raft_consensus.cc:348] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.253077 25673 raft_consensus.cc:374] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:59.253120 25673 raft_consensus.cc:729] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702c66cbc3394e6bb0ca5f0a7ebbde21, State: Initialized, Role: FOLLOWER
I20260928 06:30:59.253240 25673 consensus_queue.cc:260] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.253592 25673 ts_tablet_manager.cc:1428] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:30:59.253520 25674 raft_consensus.cc:348] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.253718 25674 raft_consensus.cc:374] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:59.253727 25388 heartbeater.cc:503] Master 127.24.11.190:41149 was elected leader, sending a full tablet report...
I20260928 06:30:59.253762 25674 raft_consensus.cc:729] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124f23086b1548b286f1beb5835aa9c0, State: Initialized, Role: FOLLOWER
I20260928 06:30:59.253878 25674 consensus_queue.cc:260] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.254043 25678 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Bootstrap starting.
I20260928 06:30:59.254153 25674 ts_tablet_manager.cc:1428] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:30:59.254266 25656 heartbeater.cc:503] Master 127.24.11.190:41149 was elected leader, sending a full tablet report...
I20260928 06:30:59.254758 25678 tablet_bootstrap.cc:654] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Neither blocks nor log segments found. Creating new log.
I20260928 06:30:59.255330 25678 log.cc:824] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Log is configured to *not* fsync() on all Append() calls
I20260928 06:30:59.256044 25678 tablet_bootstrap.cc:492] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: No bootstrap required, opened a new log
I20260928 06:30:59.256127 25678 ts_tablet_manager.cc:1397] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:30:59.257593 25678 raft_consensus.cc:348] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.257771 25678 raft_consensus.cc:374] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:30:59.257817 25678 raft_consensus.cc:729] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df47c31099c0474283ee52bcee875f60, State: Initialized, Role: FOLLOWER
I20260928 06:30:59.257941 25678 consensus_queue.cc:260] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.258225 25678 ts_tablet_manager.cc:1428] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:30:59.258283 25522 heartbeater.cc:503] Master 127.24.11.190:41149 was elected leader, sending a full tablet report...
W20260928 06:30:59.274420 25523 tablet.cc:2369] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:30:59.318451 25677 raft_consensus.cc:482] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:30:59.318578 25677 raft_consensus.cc:504] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.319032 25677 leader_election.cc:288] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df47c31099c0474283ee52bcee875f60 (127.24.11.130:40627), 124f23086b1548b286f1beb5835aa9c0 (127.24.11.131:38299)
I20260928 06:30:59.323256 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.319373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:55031 (local address 127.24.11.130:40627)
0928 06:30:59.319418 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.319419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.319428 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:30:59.319459 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.319460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.319467 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.319561 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:59.319788 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.320540 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322263 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.322862 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322949 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.322965 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.323140 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.323182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:30:59.323257 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.319306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:49483 (local address 127.24.11.131:38299)
0928 06:30:59.319354 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.319355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.319544 (+ 189us) server_negotiation.cc:407] Connection header received
0928 06:30:59.319571 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.319572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.319581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.319639 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:30:59.319827 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.320529 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322279 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.322860 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322949 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.322959 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.323140 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.323182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:30:59.323268 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.319231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40627 (local address 127.24.11.129:55031)
0928 06:30:59.319289 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.319301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.319343 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.319590 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.319591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.319593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:59.319688 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.319690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.320643 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.320644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.322119 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.322123 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.322961 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.323054 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.323056 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.323117 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.323168 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":121}
I20260928 06:30:59.323274 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.319231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:38299 (local address 127.24.11.129:49483)
0928 06:30:59.319455 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:30:59.319463 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:30:59.319504 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:30:59.319651 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:30:59.319651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:30:59.319653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:30:59.319713 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.319714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.320643 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.320644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.322120 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:30:59.322123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.322961 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.322962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:30:59.323048 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.323053 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:30:59.323117 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:30:59.323168 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":91,"thread_start_us":139,"threads_started":1}
I20260928 06:30:59.324039 25476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41813d5bc4a5449686ec69cf31d3166a" candidate_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47c31099c0474283ee52bcee875f60" is_pre_election: true
I20260928 06:30:59.324051 25610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41813d5bc4a5449686ec69cf31d3166a" candidate_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124f23086b1548b286f1beb5835aa9c0" is_pre_election: true
I20260928 06:30:59.324203 25610 raft_consensus.cc:2454] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 702c66cbc3394e6bb0ca5f0a7ebbde21 in term 0.
I20260928 06:30:59.324218 25476 raft_consensus.cc:2454] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 702c66cbc3394e6bb0ca5f0a7ebbde21 in term 0.
I20260928 06:30:59.324503 25275 leader_election.cc:302] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 702c66cbc3394e6bb0ca5f0a7ebbde21, df47c31099c0474283ee52bcee875f60; no voters:
I20260928 06:30:59.324653 25677 raft_consensus.cc:2781] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:30:59.324720 25677 raft_consensus.cc:482] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:30:59.324752 25677 raft_consensus.cc:3037] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:59.325616 25677 raft_consensus.cc:504] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.325776 25677 leader_election.cc:288] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [CANDIDATE]: Term 1 election: Requested vote from peers df47c31099c0474283ee52bcee875f60 (127.24.11.130:40627), 124f23086b1548b286f1beb5835aa9c0 (127.24.11.131:38299)
I20260928 06:30:59.325961 25476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41813d5bc4a5449686ec69cf31d3166a" candidate_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df47c31099c0474283ee52bcee875f60"
I20260928 06:30:59.325965 25610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41813d5bc4a5449686ec69cf31d3166a" candidate_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124f23086b1548b286f1beb5835aa9c0"
I20260928 06:30:59.326043 25476 raft_consensus.cc:3037] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:59.326068 25610 raft_consensus.cc:3037] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:30:59.326778 25476 raft_consensus.cc:2454] T 41813d5bc4a5449686ec69cf31d3166a P df47c31099c0474283ee52bcee875f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 702c66cbc3394e6bb0ca5f0a7ebbde21 in term 1.
I20260928 06:30:59.326891 25610 raft_consensus.cc:2454] T 41813d5bc4a5449686ec69cf31d3166a P 124f23086b1548b286f1beb5835aa9c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 702c66cbc3394e6bb0ca5f0a7ebbde21 in term 1.
I20260928 06:30:59.326978 25275 leader_election.cc:302] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 702c66cbc3394e6bb0ca5f0a7ebbde21, df47c31099c0474283ee52bcee875f60; no voters:
I20260928 06:30:59.327085 25677 raft_consensus.cc:2781] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:30:59.327237 25677 raft_consensus.cc:686] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [term 1 LEADER]: Becoming Leader. State: Replica: 702c66cbc3394e6bb0ca5f0a7ebbde21, State: Running, Role: LEADER
I20260928 06:30:59.327346 25677 consensus_queue.cc:237] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } }
I20260928 06:30:59.328217 25197 catalog_manager.cc:5495] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 702c66cbc3394e6bb0ca5f0a7ebbde21 (127.24.11.129). New cstate: current_term: 1 leader_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df47c31099c0474283ee52bcee875f60" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124f23086b1548b286f1beb5835aa9c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "702c66cbc3394e6bb0ca5f0a7ebbde21" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39121 } health_report { overall_health: HEALTHY } } }
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:30:59.346098 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.337953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.24.11.129:39121)
0928 06:30:59.338026 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.338028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.338293 (+ 265us) server_negotiation.cc:407] Connection header received
0928 06:30:59.338338 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.338339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.338348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.338402 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.338954 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.339426 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.340164 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.340440 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.340502 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.342935 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.342949 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.342949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.342957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.344840 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.345178 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.345181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.345181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.345239 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.345424 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.345425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.345425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.345626 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.345877 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.345913 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.346006 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 06:30:59.361894 25389 tablet.cc:2369] T 41813d5bc4a5449686ec69cf31d3166a P 702c66cbc3394e6bb0ca5f0a7ebbde21: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:30:59.369900 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.362615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.24.11.129:39121)
0928 06:30:59.362687 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.362689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.362740 (+ 51us) server_negotiation.cc:407] Connection header received
0928 06:30:59.362801 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.362802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.362810 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.362864 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.363485 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.364008 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.364741 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.364951 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.365008 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.367370 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.367371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.367372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.367379 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.368783 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.369096 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.369097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.369097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.369145 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.369309 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.369309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.369309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.369407 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.369481 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.369768 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.369835 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:30:59.392951 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:30:59.385629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.24.11.129:39121)
0928 06:30:59.385703 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:30:59.385706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:30:59.385758 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:30:59.385825 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:30:59.385826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:30:59.385835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:30:59.385891 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:30:59.386540 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.387015 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.387761 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:30:59.387958 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:30:59.388019 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:30:59.390394 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:30:59.390396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:30:59.390396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:30:59.390412 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:30:59.391835 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.392227 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.392228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.392229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.392290 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:30:59.392500 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:30:59.392501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:30:59.392501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:30:59.392603 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:30:59.392681 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:30:59.392776 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:30:59.392848 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 06:30:59.393488 25302 tablet_service.cc:2908] Rejecting scan request for tablet 41813d5bc4a5449686ec69cf31d3166a: Uninitialized: clean time has not yet been initialized
I20260928 06:30:59.394558 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25259
I20260928 06:30:59.400475 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25393
I20260928 06:30:59.405179 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25527
I20260928 06:30:59.409389 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25164
2026-09-28T06:30:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2058 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25697](info): set up 2 sockets
Sep 28 06:30:59 dist-test-slave-38pg krb5kdc[25697](info): commencing operation
krb5kdc: starting...
W20260928 06:31:00.497731 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 06:31:00.710572 24622 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.213s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 06:31:00.837157 24622 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.126s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:00 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:00.916268 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44223
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:44223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:00.997732 25710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:00.997951 25710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:00.997972 25710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:00.999104 25710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:00.999145 25710 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:31:00.999161 25710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:00.999173 25710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:00.999230 25710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:00.999255 25710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:01.000763 25710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:44223
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:01.001036 25710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:01.001226 25710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:01.006139 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.006201 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.006139 25718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:01.006327 25710 server_base.cc:1034] running on GCE node
I20260928 06:31:01.006671 25710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:01.006979 25710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:01.008163 25710 hybrid_clock.cc:648] HybridClock initialized: now 1790577061008148 us; error 44 us; skew 500 ppm
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:01.015309 25710 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:01.015823 25710 webserver.cc:460] Webserver started at http://127.24.11.190:41523/ using document root <none> and password file <none>
I20260928 06:31:01.016067 25710 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:01.016140 25710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:01.016296 25710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:01.017417 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "1af42299a821819b15751b1024764f4d"
server_key_iv: "e828226432c278a196159bec32f0e497"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.017846 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "1af42299a821819b15751b1024764f4d"
server_key_iv: "e828226432c278a196159bec32f0e497"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.019173 25710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:01.019878 25725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.020049 25710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:01.020154 25710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "1af42299a821819b15751b1024764f4d"
server_key_iv: "e828226432c278a196159bec32f0e497"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.020226 25710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:01.041950 25710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:01.044363 25710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:01.044536 25710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:01.049213 25710 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:44223
I20260928 06:31:01.049266 25787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:44223 every 8 connection(s)
I20260928 06:31:01.049661 25710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:01.050279 25788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:01.051255 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25710
I20260928 06:31:01.051362 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:01.051542 24622 external_mini_cluster.cc:1428] Setting key 30de08b3820babb13f5f313a0e5c6567
I20260928 06:31:01.053403 25788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Bootstrap starting.
I20260928 06:31:01.054016 25788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:01.054317 25788 log.cc:824] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:01.054996 25788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: No bootstrap required, opened a new log
I20260928 06:31:01.056305 25788 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } }
I20260928 06:31:01.056466 25788 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:01.056516 25788 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5a0b7d8a5d418b8bfe19919a58cf28, State: Initialized, Role: FOLLOWER
I20260928 06:31:01.056659 25788 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } }
I20260928 06:31:01.056740 25788 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:01.056784 25788 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:01.056833 25788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:01.057344 25788 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } }
I20260928 06:31:01.057461 25788 leader_election.cc:302] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5a0b7d8a5d418b8bfe19919a58cf28; no voters:
I20260928 06:31:01.057667 25788 leader_election.cc:288] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:01.057693 25793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:01.057857 25793 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [term 1 LEADER]: Becoming Leader. State: Replica: ff5a0b7d8a5d418b8bfe19919a58cf28, State: Running, Role: LEADER
I20260928 06:31:01.057947 25793 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } }
I20260928 06:31:01.058005 25788 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:01.058266 25793 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } } }
I20260928 06:31:01.058334 25793 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:01.058463 25793 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5a0b7d8a5d418b8bfe19919a58cf28. Latest consensus state: current_term: 1 leader_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5a0b7d8a5d418b8bfe19919a58cf28" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44223 } } }
I20260928 06:31:01.058533 25793 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:01.058621 25797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:01.059208 25797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:01.061312 25797 catalog_manager.cc:1347] Generated new cluster ID: 65ba61f0abff422995c3e23f6e5c80ba
I20260928 06:31:01.061381 25797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:01.064484 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.052086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.24.11.190:44223)
0928 06:31:01.052284 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.052293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.052318 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:01.052730 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.052736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.052783 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.052936 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.053633 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.054644 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.055450 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.057508 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.057660 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.060690 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.060694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.060695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.060706 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.062732 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.063154 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.063157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.063158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.063270 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.063490 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.063491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.063491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.063805 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.064039 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.064111 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.064252 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260928 06:31:01.067497 25797 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:01.067993 25797 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:01.069737 25797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Generated new TSK 0
I20260928 06:31:01.069896 25797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:01.096730 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44223
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:01.178184 25814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:01.178426 25814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:01.178468 25814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:01.179642 25814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:01.179713 25814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:01.179746 25814 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:01.179783 25814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:01.181246 25814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:01.181550 25814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:01.181815 25814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:01.187358 25820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.187374 25819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.187386 25822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:01.187626 25814 server_base.cc:1034] running on GCE node
I20260928 06:31:01.187876 25814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:01.188114 25814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:01.189284 25814 hybrid_clock.cc:648] HybridClock initialized: now 1790577061189261 us; error 44 us; skew 500 ppm
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:01.191983 25814 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:01.192426 25814 webserver.cc:460] Webserver started at http://127.24.11.129:39047/ using document root <none> and password file <none>
I20260928 06:31:01.192673 25814 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:01.192754 25814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:01.192904 25814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:01.194052 25814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "33c5d13d7b304155a45ca63725dfc2d5"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "f59c330d40b2f7f8bc220d99a3d360fc"
server_key_iv: "3f4ba933f5e6aa5a55debca5fdd0e40a"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.194427 25814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "33c5d13d7b304155a45ca63725dfc2d5"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "f59c330d40b2f7f8bc220d99a3d360fc"
server_key_iv: "3f4ba933f5e6aa5a55debca5fdd0e40a"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.195791 25814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:01.196475 25829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.196636 25814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:01.196727 25814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "33c5d13d7b304155a45ca63725dfc2d5"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "f59c330d40b2f7f8bc220d99a3d360fc"
server_key_iv: "3f4ba933f5e6aa5a55debca5fdd0e40a"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.196818 25814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:01.205320 25814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:01.207759 25814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:01.207926 25814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:01.208442 25814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:01.208508 25814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.208556 25814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:01.208590 25814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.216086 25814 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:42355
I20260928 06:31:01.216194 25945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:42355 every 8 connection(s)
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:01.216532 25814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:01.219400 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.209002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60189 (local address 127.24.11.190:44223)
0928 06:31:01.209046 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.209048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.209283 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:31:01.209886 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.209888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.209900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.209975 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.210966 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.211800 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.213528 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.213889 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.213968 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.216996 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.216998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.216998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.217006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.218257 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.218631 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.218631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.218632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.218682 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.218889 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.218889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.218889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.218992 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.219041 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.219258 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.219338 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:31:01.219559 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.208987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.129:60189)
0928 06:31:01.209162 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.209197 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.209780 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.210086 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.210091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.210514 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.210836 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.210844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.211910 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.211912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.213412 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.213416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.213987 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.213989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.214183 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.214869 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.214881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.216877 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.218333 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.218338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.218341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.218563 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.218738 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.218739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.218739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.218836 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.219044 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.219047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.219158 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.219241 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.219361 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260928 06:31:01.220222 25946 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44223
I20260928 06:31:01.220336 25946 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:01.220515 25946 heartbeater.cc:511] Master 127.24.11.190:44223 requested a full tablet report, sending...
I20260928 06:31:01.221036 25741 ts_manager.cc:194] Registered new tserver with Master: 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355)
I20260928 06:31:01.221915 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:60189
I20260928 06:31:01.222003 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25814
I20260928 06:31:01.222110 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:01.222304 24622 external_mini_cluster.cc:1428] Setting key dfb619276a98ddd2960827b389f94ad6
I20260928 06:31:01.227953 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.221297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.129:57179)
0928 06:31:01.221484 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.221494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.221602 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.221734 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.221735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.221957 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.222028 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.222031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.222897 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.222898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.223838 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.223841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.224244 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.224246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.224345 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.224897 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.224902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.225196 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.227010 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.227011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.227012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.227177 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.227391 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.227392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.227392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.227452 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.227721 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.227722 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.227757 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.227807 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.227882 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":138,"threads_started":1}
I20260928 06:31:01.228051 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.221338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:57179 (local address 127.24.11.190:44223)
0928 06:31:01.221397 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.221399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.221615 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:31:01.221658 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.221659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.221668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.221730 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.222098 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.222803 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.223911 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.224141 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.224205 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.225287 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.225289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.225289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.225297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.226919 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.227250 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.227251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.227252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.227314 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.227513 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.227514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.227515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.227656 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.227733 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.227838 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.227920 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:01.248202 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44223
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:01.328552 25955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:01.328747 25955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:01.328769 25955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:01.330024 25955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:01.330065 25955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:01.330087 25955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:01.330128 25955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:01.331637 25955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:01.331888 25955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:01.332123 25955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:01.337122 25961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.337160 25960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.337118 25963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:01.337327 25955 server_base.cc:1034] running on GCE node
I20260928 06:31:01.337661 25955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:01.337953 25955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:01.339125 25955 hybrid_clock.cc:648] HybridClock initialized: now 1790577061339111 us; error 31 us; skew 500 ppm
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:01.341637 25955 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:01.342160 25955 webserver.cc:460] Webserver started at http://127.24.11.130:33297/ using document root <none> and password file <none>
I20260928 06:31:01.342414 25955 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:01.342489 25955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:01.342643 25955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:01.343693 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "2366aa34ca1f4894aa5734518c14fda1"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "c18dd1e12998da9590dc0a6705e672e1"
server_key_iv: "99b8491c706ece3ba912690338e58952"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.344064 25955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "2366aa34ca1f4894aa5734518c14fda1"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "c18dd1e12998da9590dc0a6705e672e1"
server_key_iv: "99b8491c706ece3ba912690338e58952"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.345520 25955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:01.346458 25970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.346663 25955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:01.346767 25955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "2366aa34ca1f4894aa5734518c14fda1"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "c18dd1e12998da9590dc0a6705e672e1"
server_key_iv: "99b8491c706ece3ba912690338e58952"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.346886 25955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:01.376371 25955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:01.378372 25955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:01.378548 25955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:01.379132 25955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:01.379197 25955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.379240 25955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:01.379262 25955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:01.387039 25955 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:39529
I20260928 06:31:01.387180 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:39529 every 8 connection(s)
I20260928 06:31:01.387507 25955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:01.389185 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.379615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35163 (local address 127.24.11.190:44223)
0928 06:31:01.379688 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.379690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.379947 (+ 257us) server_negotiation.cc:407] Connection header received
0928 06:31:01.380673 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.380674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.380684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.380725 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.381747 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.382258 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.383503 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.383763 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.383848 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.386259 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.386262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.386262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.386271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.388045 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.388412 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.388413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.388413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.388465 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.388639 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.388639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.388640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.388744 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.388795 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.389061 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.389123 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:01.389393 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.379586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.130:35163)
0928 06:31:01.379797 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.379846 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.380564 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.380826 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.380831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.381258 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.381621 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.381630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.382360 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.382361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.383414 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.383417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.383870 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.383871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.384011 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.384549 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.384559 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.386170 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.388124 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.388128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.388130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.388346 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.388506 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.388506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.388506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.388589 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.388797 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.388802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.388951 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.389035 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.389155 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260928 06:31:01.390035 26086 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44223
I20260928 06:31:01.390123 26086 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:01.390311 26086 heartbeater.cc:511] Master 127.24.11.190:44223 requested a full tablet report, sending...
I20260928 06:31:01.390682 25742 ts_manager.cc:194] Registered new tserver with Master: 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529)
I20260928 06:31:01.391156 25742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:35163
I20260928 06:31:01.393246 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 25955
I20260928 06:31:01.393354 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:01.393570 24622 external_mini_cluster.cc:1428] Setting key eba7fbcb03b2f0bfbaf6204d2fcc58cb
I20260928 06:31:01.397318 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.391138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:58611 (local address 127.24.11.190:44223)
0928 06:31:01.391176 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.391178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.391348 (+ 170us) server_negotiation.cc:407] Connection header received
0928 06:31:01.391368 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.391368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.391377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.391422 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.391750 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.392228 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.392936 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.393219 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.393287 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.394357 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.394359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.394359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.394367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.395957 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.396464 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.396465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.396465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.396515 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.396746 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.396747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.396747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.396947 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.397019 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.397182 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.397250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:01.397450 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.391065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.130:58611)
0928 06:31:01.391221 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.391232 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.391300 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.391425 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.391426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.391613 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.391672 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.391673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.392332 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.392333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.392840 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.392841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.393310 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.393311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.393371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.393860 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.393865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.394252 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.396135 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.396136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.396136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.396388 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.396578 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.396579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.396580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.396650 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.397033 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.397034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.397086 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.397162 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.397286 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:01.418663 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44223
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:01.501221 26095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:01.501432 26095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:01.501456 26095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:01.502641 26095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:01.502684 26095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:01.502705 26095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:01.502723 26095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:01.504153 26095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:44223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:01.504390 26095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:01.504637 26095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:01.509660 26103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:01.509675 26101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:01.509687 26095 server_base.cc:1034] running on GCE node
W20260928 06:31:01.509789 26100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:01.510176 26095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:01.510440 26095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:01.511612 26095 hybrid_clock.cc:648] HybridClock initialized: now 1790577061511609 us; error 33 us; skew 500 ppm
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:01.514227 26095 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:01.514664 26095 webserver.cc:460] Webserver started at http://127.24.11.131:32815/ using document root <none> and password file <none>
I20260928 06:31:01.514936 26095 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:01.515009 26095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:01.515156 26095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:01.516188 26095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "567b08db08614f7ca15efc57b9f947f4"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "e53f5ff8c2eda85e74c6cc2175099a17"
server_key_iv: "f51d9d0c024ab45a1d270d819c5c66f1"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.516568 26095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "567b08db08614f7ca15efc57b9f947f4"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "e53f5ff8c2eda85e74c6cc2175099a17"
server_key_iv: "f51d9d0c024ab45a1d270d819c5c66f1"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.517958 26095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:01.518724 26110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.518898 26095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:01.519011 26095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "567b08db08614f7ca15efc57b9f947f4"
format_stamp: "Formatted at 2026-09-28 06:31:01 on dist-test-slave-38pg"
server_key: "e53f5ff8c2eda85e74c6cc2175099a17"
server_key_iv: "f51d9d0c024ab45a1d270d819c5c66f1"
server_key_version: "encryptionkey@0"
I20260928 06:31:01.519090 26095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:01.534252 26095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:01.536262 26095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:01.536448 26095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:01.537030 26095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:01.537098 26095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:01.537149 26095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:01.537181 26095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:01.545279 26095 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:45209
I20260928 06:31:01.545363 26225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:45209 every 8 connection(s)
I20260928 06:31:01.545724 26095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:01.547086 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.537548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54641 (local address 127.24.11.190:44223)
0928 06:31:01.537594 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.537596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.537854 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:31:01.538381 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.538382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.538395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.538444 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.539166 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.539693 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.540696 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.540965 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.541051 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.543846 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.543848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.543849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.543857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.545651 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.546127 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.546129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.546129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.546190 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.546434 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.546435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.546435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.546559 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.546637 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.546906 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.546995 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:01.547264 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.537521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.131:54641)
0928 06:31:01.537734 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.537769 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.538275 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.538518 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.538522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.538875 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.539068 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.539073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.539793 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.539794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.540602 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.540604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.541059 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.541061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.541221 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.541788 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.541801 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.543747 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.545742 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.545748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.545750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.546035 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.546244 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.546244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.546245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.546356 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.546628 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.546635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.546775 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.546884 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.547041 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260928 06:31:01.547732 26226 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44223
I20260928 06:31:01.547813 26226 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:01.547987 26226 heartbeater.cc:511] Master 127.24.11.190:44223 requested a full tablet report, sending...
I20260928 06:31:01.548349 25741 ts_manager.cc:194] Registered new tserver with Master: 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131:45209)
I20260928 06:31:01.548729 25741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:54641
I20260928 06:31:01.553427 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 26095
I20260928 06:31:01.553596 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:01.553822 24622 external_mini_cluster.cc:1428] Setting key cf1575d2e8c782745eece60b5f23b03d
I20260928 06:31:01.554316 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.548878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:56009 (local address 127.24.11.190:44223)
0928 06:31:01.548914 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.548923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.549025 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:31:01.549067 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.549068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.549078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.549137 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.549581 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.550104 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.550887 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.551079 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.551131 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.551947 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.551948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.551948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.551954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.553106 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.553576 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.553578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.553578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.553639 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.553838 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.553839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.553839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.553956 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.554017 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.554155 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.554228 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:31:01.554332 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.548823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.24.11.131:56009)
0928 06:31:01.548944 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.548956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.549021 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.549178 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.549179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.549397 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:01.549468 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.549470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.550194 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.550195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.550828 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.550830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.551145 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.551146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.551203 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.551575 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:01.551579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:01.551861 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:01.553179 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.553180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.553180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.553476 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.553700 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:01.553701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:01.553702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:01.553765 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:01.554009 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:01.554010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:01.554055 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.554130 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.554231 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260928 06:31:01.555047 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:31:01 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:01.577047 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.570024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.24.11.190:44223)
0928 06:31:01.570104 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.570106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.570116 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:01.570167 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.570168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.570178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.570243 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:01.570795 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.571280 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.572024 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.572217 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.572304 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.574488 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:01.574490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:01.574490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:01.574499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:01.575898 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.576266 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.576267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.576268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.576327 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:01.576524 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:01.576525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:01.576525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:01.576640 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:01.576747 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.576862 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.576941 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:31:01.578953 25741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:01.580503 25741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:01.587690 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.582937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45209 (local address 127.0.0.1:56306)
0928 06:31:01.583218 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.583251 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.583320 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.583937 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.583938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.583940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:01.584033 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.584035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.585188 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.585188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.586209 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.586211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587364 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.587365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.587482 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.587502 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.587558 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.587609 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260928 06:31:01.588021 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.582936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:42355 (local address 127.0.0.1:57590)
0928 06:31:01.583202 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.583249 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.583343 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.583933 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.583934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.583936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:01.584018 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.584020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.585006 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.585007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.586658 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.586660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587628 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.587629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.587689 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.587693 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.587726 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.587941 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":92,"thread_start_us":50,"threads_started":1}
I20260928 06:31:01.588065 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.583379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.24.11.131:45209)
0928 06:31:01.583573 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.583578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.583588 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:01.583799 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.583802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.583821 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.583982 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:01.584098 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.585109 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.586394 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587253 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.587817 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.587887 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.587913 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.587951 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260928 06:31:01.588080 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.583191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.24.11.130:39529)
0928 06:31:01.583383 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.583389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.583498 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:31:01.583564 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.583566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.583584 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.583670 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:01.583851 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.584994 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.586580 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587398 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.587477 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.587539 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.587905 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.587959 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260928 06:31:01.588806 26160 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (DEFAULT_TABLE table=test-table [id=121acdcdfa5248ec9e2771f982baae3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:01.589175 26160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:01.588259 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.582937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39529 (local address 127.0.0.1:51232)
0928 06:31:01.583430 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.583439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.583489 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.583683 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.583686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.583689 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:01.583760 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.583761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.585064 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.585065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.586484 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.586487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587577 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.587578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.587648 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.587653 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.587704 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.587752 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":68,"thread_start_us":93,"threads_started":1}
I20260928 06:31:01.590021 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.583057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.24.11.129:42355)
0928 06:31:01.583754 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.583759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.583768 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:01.583805 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.583807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.583824 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.584103 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:01.584168 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.584908 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.586750 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.587535 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.589682 (+ 2147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.589802 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.589830 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.589869 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":79,"threads_started":1}
I20260928 06:31:01.590363 26020 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (DEFAULT_TABLE table=test-table [id=121acdcdfa5248ec9e2771f982baae3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:01.590735 26020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:01.590651 25880 tablet_service.cc:1467] Processing CreateTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (DEFAULT_TABLE table=test-table [id=121acdcdfa5248ec9e2771f982baae3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:01.591004 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:01.592736 26249 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Bootstrap starting.
I20260928 06:31:01.593756 26249 tablet_bootstrap.cc:654] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:01.594125 26250 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Bootstrap starting.
I20260928 06:31:01.594585 26251 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Bootstrap starting.
I20260928 06:31:01.594622 26249 log.cc:824] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:01.594949 26250 tablet_bootstrap.cc:654] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:01.595286 26251 tablet_bootstrap.cc:654] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:01.595305 26250 log.cc:824] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:01.595435 26249 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: No bootstrap required, opened a new log
I20260928 06:31:01.595503 26249 ts_tablet_manager.cc:1397] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:31:01.595703 26251 log.cc:824] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:01.596096 26250 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: No bootstrap required, opened a new log
I20260928 06:31:01.596186 26250 ts_tablet_manager.cc:1397] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:01.596551 26251 tablet_bootstrap.cc:492] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: No bootstrap required, opened a new log
I20260928 06:31:01.596661 26251 ts_tablet_manager.cc:1397] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:01.597259 26249 raft_consensus.cc:348] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.597416 26249 raft_consensus.cc:374] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:01.597483 26249 raft_consensus.cc:729] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567b08db08614f7ca15efc57b9f947f4, State: Initialized, Role: FOLLOWER
I20260928 06:31:01.597648 26249 consensus_queue.cc:260] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.597955 26249 ts_tablet_manager.cc:1428] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:01.597966 26226 heartbeater.cc:503] Master 127.24.11.190:44223 was elected leader, sending a full tablet report...
I20260928 06:31:01.598439 26250 raft_consensus.cc:348] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.598616 26250 raft_consensus.cc:374] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:01.598706 26250 raft_consensus.cc:729] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c5d13d7b304155a45ca63725dfc2d5, State: Initialized, Role: FOLLOWER
I20260928 06:31:01.598878 26250 consensus_queue.cc:260] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.598899 26251 raft_consensus.cc:348] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.599126 26251 raft_consensus.cc:374] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:01.599251 26250 ts_tablet_manager.cc:1428] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:01.599295 25946 heartbeater.cc:503] Master 127.24.11.190:44223 was elected leader, sending a full tablet report...
I20260928 06:31:01.599221 26251 raft_consensus.cc:729] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2366aa34ca1f4894aa5734518c14fda1, State: Initialized, Role: FOLLOWER
I20260928 06:31:01.599694 26251 consensus_queue.cc:260] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.600176 26251 ts_tablet_manager.cc:1428] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:01.600199 26086 heartbeater.cc:503] Master 127.24.11.190:44223 was elected leader, sending a full tablet report...
W20260928 06:31:01.638077 26087 tablet.cc:2369] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:01.717375 25947 tablet.cc:2369] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:01.796360 26227 tablet.cc:2369] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:01.965206 26256 raft_consensus.cc:482] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:01.965363 26256 raft_consensus.cc:504] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.965850 26256 leader_election.cc:288] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131:45209), 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529)
I20260928 06:31:01.970049 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.966157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:35445 (local address 127.24.11.130:39529)
0928 06:31:01.966206 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.966209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.966301 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:31:01.966356 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.966357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.966367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.966448 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:01.966631 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.967392 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969040 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.969658 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969742 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.969751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.969937 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.969982 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:31:01.970049 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.966161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:56113 (local address 127.24.11.131:45209)
0928 06:31:01.966207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:01.966209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:01.966296 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:31:01.966357 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:01.966357 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:01.966367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:01.966448 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:01.966631 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.967392 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969039 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.969670 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969742 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.969751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:01.969938 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 06:31:01.969978 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:01.970063 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.966058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39529 (local address 127.24.11.129:35445)
0928 06:31:01.966202 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.966214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.966290 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.966476 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.966476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.966480 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:01.966552 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.966554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.967486 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.967487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.968946 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.968948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.969755 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.969841 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.969848 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.969911 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.969964 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":93,"spinlock_wait_cycles":16640,"thread_start_us":81,"threads_started":1}
I20260928 06:31:01.970077 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:01.966058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45209 (local address 127.24.11.129:56113)
0928 06:31:01.966202 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:01.966214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:01.966290 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:01.966476 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:01.966476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:01.966480 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:01.966560 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.966562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.967488 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.967489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.968921 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:01.968925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:01.969754 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:01.969755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:01.969841 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:01.969849 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:31:01.969911 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:31:01.969958 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":111,"thread_start_us":75,"threads_started":1}
I20260928 06:31:01.970424 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f6a63bd9740b8a58f7fb4a8f50340" candidate_uuid: "33c5d13d7b304155a45ca63725dfc2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567b08db08614f7ca15efc57b9f947f4" is_pre_election: true
I20260928 06:31:01.970597 26180 raft_consensus.cc:2454] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c5d13d7b304155a45ca63725dfc2d5 in term 0.
I20260928 06:31:01.970716 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f6a63bd9740b8a58f7fb4a8f50340" candidate_uuid: "33c5d13d7b304155a45ca63725dfc2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1" is_pre_election: true
I20260928 06:31:01.970886 26040 raft_consensus.cc:2454] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33c5d13d7b304155a45ca63725dfc2d5 in term 0.
I20260928 06:31:01.970904 25832 leader_election.cc:302] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c5d13d7b304155a45ca63725dfc2d5, 567b08db08614f7ca15efc57b9f947f4; no voters:
I20260928 06:31:01.971107 26256 raft_consensus.cc:2781] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:01.971165 26256 raft_consensus.cc:482] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:01.971182 26256 raft_consensus.cc:3037] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:01.971910 26256 raft_consensus.cc:504] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.972059 26256 leader_election.cc:288] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [CANDIDATE]: Term 1 election: Requested vote from peers 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131:45209), 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529)
I20260928 06:31:01.972208 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f6a63bd9740b8a58f7fb4a8f50340" candidate_uuid: "33c5d13d7b304155a45ca63725dfc2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1"
I20260928 06:31:01.972231 26180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c8f6a63bd9740b8a58f7fb4a8f50340" candidate_uuid: "33c5d13d7b304155a45ca63725dfc2d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567b08db08614f7ca15efc57b9f947f4"
I20260928 06:31:01.972302 26040 raft_consensus.cc:3037] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:01.972308 26180 raft_consensus.cc:3037] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:01.973026 26040 raft_consensus.cc:2454] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c5d13d7b304155a45ca63725dfc2d5 in term 1.
I20260928 06:31:01.973057 26180 raft_consensus.cc:2454] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33c5d13d7b304155a45ca63725dfc2d5 in term 1.
I20260928 06:31:01.973248 25830 leader_election.cc:302] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2366aa34ca1f4894aa5734518c14fda1, 33c5d13d7b304155a45ca63725dfc2d5; no voters:
I20260928 06:31:01.973384 26256 raft_consensus.cc:2781] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:01.973578 26256 raft_consensus.cc:686] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 LEADER]: Becoming Leader. State: Replica: 33c5d13d7b304155a45ca63725dfc2d5, State: Running, Role: LEADER
I20260928 06:31:01.973675 26256 consensus_queue.cc:237] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:01.974502 25741 catalog_manager.cc:5495] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129). New cstate: current_term: 1 leader_uuid: "33c5d13d7b304155a45ca63725dfc2d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:02.028198 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.025665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.24.11.129:42355)
0928 06:31:02.025732 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.025734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.025743 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:02.025789 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.025790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.025807 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.025883 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:02.026050 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.026534 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.027346 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.027554 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.027632 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.027845 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.027949 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.028043 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.028085 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.028108 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:02.030027 26040 raft_consensus.cc:1261] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Refusing update from remote peer 33c5d13d7b304155a45ca63725dfc2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.030028 26180 raft_consensus.cc:1261] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Refusing update from remote peer 33c5d13d7b304155a45ca63725dfc2d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.030514 26256 consensus_queue.cc:1035] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.030665 26260 consensus_queue.cc:1035] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.033491 26265 mvcc.cc:205] Tried to move back new op lower bound from 7334203646072164352 to 7334203645844717568. Current Snapshot: MvccSnapshot[applied={T|T < 7334203646072164352}]
I20260928 06:31:02.033699 26263 mvcc.cc:205] Tried to move back new op lower bound from 7334203646072164352 to 7334203645844717568. Current Snapshot: MvccSnapshot[applied={T|T < 7334203646072164352 or (T in {7334203646072164352})}]
I20260928 06:31:02.036635 25741 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:02.036738 25741 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:02.037580 25741 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Sending DeleteTablet for 3 replicas of tablet 6c8f6a63bd9740b8a58f7fb4a8f50340
I20260928 06:31:02.038018 26020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:51232
I20260928 06:31:02.038226 26272 tablet_replica.cc:330] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: stopping tablet replica
I20260928 06:31:02.038264 25880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57590
I20260928 06:31:02.038349 26272 raft_consensus.cc:2229] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:02.038460 26272 raft_consensus.cc:2258] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.038533 25741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:31:02.038620 26160 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56306
W20260928 06:31:02.038684 25741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:02.039062 26274 tablet_replica.cc:330] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: stopping tablet replica
I20260928 06:31:02.039122 26272 ts_tablet_manager.cc:1905] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.039149 26274 raft_consensus.cc:2229] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:02.039240 26274 raft_consensus.cc:2258] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.039693 26274 ts_tablet_manager.cc:1905] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.040266 25880 tablet_service.cc:1467] Processing CreateTablet for tablet f8a411cff909456cb24e2fde8124a7f2 (DEFAULT_TABLE table=test-table [id=9161be98e44b4f5b838652b640b9fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:02.040376 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a411cff909456cb24e2fde8124a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.040983 26272 ts_tablet_manager.cc:1918] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:02.041059 26272 log.cc:1196] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/6c8f6a63bd9740b8a58f7fb4a8f50340
I20260928 06:31:02.041363 26274 ts_tablet_manager.cc:1918] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:02.041426 26274 log.cc:1196] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/6c8f6a63bd9740b8a58f7fb4a8f50340
I20260928 06:31:02.041491 26272 ts_tablet_manager.cc:1939] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 2366aa34ca1f4894aa5734518c14fda1: Deleting consensus metadata
I20260928 06:31:02.041716 26274 ts_tablet_manager.cc:1939] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 567b08db08614f7ca15efc57b9f947f4: Deleting consensus metadata
I20260928 06:31:02.042236 26250 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Bootstrap starting.
I20260928 06:31:02.042387 25726 catalog_manager.cc:4841] TS 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529): tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (table test-table [id=121acdcdfa5248ec9e2771f982baae3f]) successfully deleted
I20260928 06:31:02.042532 26020 tablet_service.cc:1467] Processing CreateTablet for tablet f8a411cff909456cb24e2fde8124a7f2 (DEFAULT_TABLE table=test-table [id=9161be98e44b4f5b838652b640b9fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:02.042621 26020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a411cff909456cb24e2fde8124a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.042682 26250 tablet_bootstrap.cc:654] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.043462 26250 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: No bootstrap required, opened a new log
I20260928 06:31:02.043501 26250 ts_tablet_manager.cc:1397] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:02.043637 26250 raft_consensus.cc:348] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.043717 26250 raft_consensus.cc:374] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.043742 26250 raft_consensus.cc:729] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c5d13d7b304155a45ca63725dfc2d5, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.043802 26250 consensus_queue.cc:260] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.043881 26251 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Bootstrap starting.
I20260928 06:31:02.043906 26250 ts_tablet_manager.cc:1428] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:02.044225 26251 tablet_bootstrap.cc:654] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.044843 25728 catalog_manager.cc:4841] TS 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131:45209): tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (table test-table [id=121acdcdfa5248ec9e2771f982baae3f]) successfully deleted
I20260928 06:31:02.044909 26160 tablet_service.cc:1467] Processing CreateTablet for tablet f8a411cff909456cb24e2fde8124a7f2 (DEFAULT_TABLE table=test-table [id=9161be98e44b4f5b838652b640b9fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:02.045017 26160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8a411cff909456cb24e2fde8124a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.045060 26251 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: No bootstrap required, opened a new log
I20260928 06:31:02.045092 26251 ts_tablet_manager.cc:1397] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:02.045193 26251 raft_consensus.cc:348] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.045238 26251 raft_consensus.cc:374] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.045250 26251 raft_consensus.cc:729] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2366aa34ca1f4894aa5734518c14fda1, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.045320 26251 consensus_queue.cc:260] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.045451 26251 ts_tablet_manager.cc:1428] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:02.046317 26249 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Bootstrap starting.
I20260928 06:31:02.046707 26249 tablet_bootstrap.cc:654] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.047039 26273 tablet_replica.cc:330] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: stopping tablet replica
I20260928 06:31:02.047170 26273 raft_consensus.cc:2229] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:02.047304 26273 raft_consensus.cc:2258] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.047312 26249 tablet_bootstrap.cc:492] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: No bootstrap required, opened a new log
I20260928 06:31:02.047358 26249 ts_tablet_manager.cc:1397] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:02.047557 26249 raft_consensus.cc:348] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.047631 26249 raft_consensus.cc:374] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.047657 26249 raft_consensus.cc:729] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567b08db08614f7ca15efc57b9f947f4, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.047726 26249 consensus_queue.cc:260] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.047823 26249 ts_tablet_manager.cc:1428] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:02.048418 26273 ts_tablet_manager.cc:1905] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.050082 26273 ts_tablet_manager.cc:1918] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:02.050225 26273 log.cc:1196] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/6c8f6a63bd9740b8a58f7fb4a8f50340
I20260928 06:31:02.050527 26273 ts_tablet_manager.cc:1939] T 6c8f6a63bd9740b8a58f7fb4a8f50340 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting consensus metadata
I20260928 06:31:02.050948 25729 catalog_manager.cc:4841] TS 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355): tablet 6c8f6a63bd9740b8a58f7fb4a8f50340 (table test-table [id=121acdcdfa5248ec9e2771f982baae3f]) successfully deleted
I20260928 06:31:02.322820 26255 raft_consensus.cc:482] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:02.322944 26255 raft_consensus.cc:504] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.323401 26255 leader_election.cc:288] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355), 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529)
I20260928 06:31:02.327649 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.323811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40173 (local address 127.24.11.129:42355)
0928 06:31:02.323875 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.323876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.323886 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:02.323956 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.323957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.323970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.324028 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:02.324209 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.324966 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.326669 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.327230 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.327318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.327327 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.327527 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.327579 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:02.327649 26247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.323708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:53937 (local address 127.24.11.130:39529)
0928 06:31:02.323767 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.323769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.323918 (+ 149us) server_negotiation.cc:407] Connection header received
0928 06:31:02.323950 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.323951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.323962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.324022 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:02.324209 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.324969 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.326669 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.327230 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.327318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.327327 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.327533 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.327576 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:31:02.327657 26279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.323612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39529 (local address 127.24.11.131:53937)
0928 06:31:02.323744 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:02.323756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:02.323856 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:02.324047 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:02.324048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:02.324050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:02.324131 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.324132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.325075 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.325077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.326566 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.326569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.327335 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.327335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.327409 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.327426 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:31:02.327480 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:31:02.327534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":145,"thread_start_us":68,"threads_started":1}
I20260928 06:31:02.327657 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.323612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:42355 (local address 127.24.11.131:40173)
0928 06:31:02.323833 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:02.323843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:02.323883 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:02.324047 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:02.324048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:02.324050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:02.324130 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.324132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.325075 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.325077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.326565 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.326569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.327334 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.327335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.327409 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.327417 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:31:02.327480 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:31:02.327534 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":81,"thread_start_us":48,"threads_started":1}
I20260928 06:31:02.327967 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a411cff909456cb24e2fde8124a7f2" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1" is_pre_election: true
I20260928 06:31:02.328066 26040 raft_consensus.cc:2454] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 0.
I20260928 06:31:02.328107 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a411cff909456cb24e2fde8124a7f2" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5d13d7b304155a45ca63725dfc2d5" is_pre_election: true
I20260928 06:31:02.328245 25900 raft_consensus.cc:2454] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 0.
I20260928 06:31:02.328279 26111 leader_election.cc:302] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2366aa34ca1f4894aa5734518c14fda1, 567b08db08614f7ca15efc57b9f947f4; no voters:
I20260928 06:31:02.328433 26255 raft_consensus.cc:2781] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:02.328466 26255 raft_consensus.cc:482] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:02.328491 26255 raft_consensus.cc:3037] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.329176 26255 raft_consensus.cc:504] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.329339 26255 leader_election.cc:288] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 election: Requested vote from peers 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355), 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529)
I20260928 06:31:02.329509 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a411cff909456cb24e2fde8124a7f2" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1"
I20260928 06:31:02.329514 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8a411cff909456cb24e2fde8124a7f2" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5d13d7b304155a45ca63725dfc2d5"
I20260928 06:31:02.329634 25900 raft_consensus.cc:3037] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.329625 26040 raft_consensus.cc:3037] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.330617 25900 raft_consensus.cc:2454] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 1.
I20260928 06:31:02.330777 26040 raft_consensus.cc:2454] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 1.
I20260928 06:31:02.330859 26114 leader_election.cc:302] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c5d13d7b304155a45ca63725dfc2d5, 567b08db08614f7ca15efc57b9f947f4; no voters:
I20260928 06:31:02.330974 26255 raft_consensus.cc:2781] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:02.331120 26255 raft_consensus.cc:686] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 1 LEADER]: Becoming Leader. State: Replica: 567b08db08614f7ca15efc57b9f947f4, State: Running, Role: LEADER
I20260928 06:31:02.331215 26255 consensus_queue.cc:237] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } }
I20260928 06:31:02.331934 25742 catalog_manager.cc:5495] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } health_report { overall_health: HEALTHY } } }
Sep 28 06:31:02 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:02.433794 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.426399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.24.11.190:44223)
0928 06:31:02.426538 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.426541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.426633 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:31:02.426730 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.426730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.426742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.426826 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:02.427409 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.427913 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.428607 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.428794 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.428876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.431143 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:02.431145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:02.431145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:02.431153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:02.432636 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.432987 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.432988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.432988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.433042 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.433266 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.433267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.433267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.433368 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:02.433428 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.433642 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.433717 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 06:31:02.433996 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.426337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44223 (local address 127.0.0.1:52286)
0928 06:31:02.426547 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:02.426559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:02.426654 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:02.426845 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:02.426845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:02.427266 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:02.427314 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.427316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.427997 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.427998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.428513 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.428514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.428879 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.428879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.428941 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.429491 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:02.429496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:02.431006 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:02.432711 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:02.432718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:02.432720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:02.432922 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:02.433131 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:02.433132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:02.433133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:02.433200 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:02.433428 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:02.433434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:02.433480 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:31:02.433628 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 06:31:02.433837 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":67,"thread_start_us":101,"threads_started":1}
I20260928 06:31:02.436043 25742 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:52286:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:31:02.436261 25742 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:02.437844 26020 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7d61c761684c808e4dbd5dabc64b2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89261f19dda94f788106e04a2680e1fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:02.437846 26160 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7d61c761684c808e4dbd5dabc64b2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89261f19dda94f788106e04a2680e1fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:02.438017 26020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7d61c761684c808e4dbd5dabc64b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.438016 26160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7d61c761684c808e4dbd5dabc64b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.439224 26251 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: Bootstrap starting.
I20260928 06:31:02.439244 26249 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: Bootstrap starting.
I20260928 06:31:02.439371 25880 tablet_service.cc:1467] Processing CreateTablet for tablet 5f7d61c761684c808e4dbd5dabc64b2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89261f19dda94f788106e04a2680e1fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:02.439558 25880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f7d61c761684c808e4dbd5dabc64b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:02.439628 26249 tablet_bootstrap.cc:654] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.439807 26251 tablet_bootstrap.cc:654] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.441066 26249 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: No bootstrap required, opened a new log
I20260928 06:31:02.441129 26249 ts_tablet_manager.cc:1397] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:02.441291 26249 raft_consensus.cc:348] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.441359 26249 raft_consensus.cc:374] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.441375 26249 raft_consensus.cc:729] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567b08db08614f7ca15efc57b9f947f4, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.441439 26249 consensus_queue.cc:260] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.441602 26250 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: Bootstrap starting.
I20260928 06:31:02.441607 26255 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.441674 26255 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.441984 26250 tablet_bootstrap.cc:654] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:02.442255 26249 ts_tablet_manager.cc:1428] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:02.442514 26251 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: No bootstrap required, opened a new log
I20260928 06:31:02.442564 26250 tablet_bootstrap.cc:492] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: No bootstrap required, opened a new log
I20260928 06:31:02.442569 26251 ts_tablet_manager.cc:1397] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:31:02.442622 26250 ts_tablet_manager.cc:1397] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:02.442726 26251 raft_consensus.cc:348] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.442751 26250 raft_consensus.cc:348] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.442826 26251 raft_consensus.cc:374] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.442843 26250 raft_consensus.cc:374] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:02.442845 26251 raft_consensus.cc:729] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2366aa34ca1f4894aa5734518c14fda1, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.442868 26250 raft_consensus.cc:729] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c5d13d7b304155a45ca63725dfc2d5, State: Initialized, Role: FOLLOWER
I20260928 06:31:02.442914 26250 consensus_queue.cc:260] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.442914 26251 consensus_queue.cc:260] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.443006 26250 ts_tablet_manager.cc:1428] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:02.443010 26251 ts_tablet_manager.cc:1428] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:02.443033 26256 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.443131 26256 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.443154 26257 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.443233 26257 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.568001 26255 raft_consensus.cc:482] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:02.568159 26255 raft_consensus.cc:504] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.568372 26255 leader_election.cc:288] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529), 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355)
I20260928 06:31:02.568632 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7d61c761684c808e4dbd5dabc64b2f" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5d13d7b304155a45ca63725dfc2d5" is_pre_election: true
I20260928 06:31:02.568634 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7d61c761684c808e4dbd5dabc64b2f" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1" is_pre_election: true
I20260928 06:31:02.568717 25900 raft_consensus.cc:2454] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 0.
I20260928 06:31:02.568747 26040 raft_consensus.cc:2454] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 0.
I20260928 06:31:02.568907 26114 leader_election.cc:302] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c5d13d7b304155a45ca63725dfc2d5, 567b08db08614f7ca15efc57b9f947f4; no voters:
I20260928 06:31:02.569013 26255 raft_consensus.cc:2781] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:02.569053 26255 raft_consensus.cc:482] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:02.569082 26255 raft_consensus.cc:3037] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.569957 26255 raft_consensus.cc:504] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.570092 26255 leader_election.cc:288] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 election: Requested vote from peers 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529), 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355)
I20260928 06:31:02.570274 25900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7d61c761684c808e4dbd5dabc64b2f" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c5d13d7b304155a45ca63725dfc2d5"
I20260928 06:31:02.570264 26040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f7d61c761684c808e4dbd5dabc64b2f" candidate_uuid: "567b08db08614f7ca15efc57b9f947f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2366aa34ca1f4894aa5734518c14fda1"
I20260928 06:31:02.570336 26040 raft_consensus.cc:3037] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.570355 25900 raft_consensus.cc:3037] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:02.570919 25900 raft_consensus.cc:2454] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 1.
I20260928 06:31:02.570933 26040 raft_consensus.cc:2454] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567b08db08614f7ca15efc57b9f947f4 in term 1.
I20260928 06:31:02.571120 26114 leader_election.cc:302] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2366aa34ca1f4894aa5734518c14fda1, 33c5d13d7b304155a45ca63725dfc2d5, 567b08db08614f7ca15efc57b9f947f4; no voters:
I20260928 06:31:02.571244 26255 raft_consensus.cc:2781] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:02.571318 26255 raft_consensus.cc:686] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [term 1 LEADER]: Becoming Leader. State: Replica: 567b08db08614f7ca15efc57b9f947f4, State: Running, Role: LEADER
I20260928 06:31:02.571370 26255 consensus_queue.cc:237] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } }
I20260928 06:31:02.571656 26280 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: TxnStatusTablet state changed. Reason: New leader 567b08db08614f7ca15efc57b9f947f4. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.571754 26280 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:02.571916 26293 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:31:02.572018 26293 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:31:02.572037 25740 catalog_manager.cc:5495] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:02.589650 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.586758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45209 (local address 127.0.0.1:42602)
0928 06:31:02.586852 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:02.586864 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:02.586913 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:02.587268 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:02.587269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:02.587270 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:02.587368 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.587371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.588078 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.588078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.588659 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.588661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.589048 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.589050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.589109 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.589123 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.589432 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.589442 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:31:02.589487 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:31:02.589576 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:31:02.589651 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.586989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42602 (local address 127.24.11.131:45209)
0928 06:31:02.587107 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.587109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.587116 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:02.587160 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.587161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.587176 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.587262 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:02.587451 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.587986 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.588746 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.588964 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.589038 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.589223 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.589334 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.589438 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.589489 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.589514 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260928 06:31:02.591123 25900 raft_consensus.cc:1261] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Refusing update from remote peer 567b08db08614f7ca15efc57b9f947f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.591140 26040 raft_consensus.cc:1261] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Refusing update from remote peer 567b08db08614f7ca15efc57b9f947f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.591409 26280 consensus_queue.cc:1035] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.591522 26255 consensus_queue.cc:1035] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.592079 26257 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: TxnStatusTablet state changed. Reason: New leader 567b08db08614f7ca15efc57b9f947f4. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.592185 26257 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.592288 26256 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: TxnStatusTablet state changed. Reason: New leader 567b08db08614f7ca15efc57b9f947f4. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.592381 26256 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.592810 26255 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.592895 26255 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:02.593178 26256 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.593258 26256 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 33c5d13d7b304155a45ca63725dfc2d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.594578 26280 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.594669 26280 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 567b08db08614f7ca15efc57b9f947f4: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:02.595229 26257 tablet_replica.cc:439] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "567b08db08614f7ca15efc57b9f947f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 } } peers { permanent_uuid: "567b08db08614f7ca15efc57b9f947f4" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45209 } } peers { permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 } } }
I20260928 06:31:02.595316 26257 tablet_replica.cc:442] T 5f7d61c761684c808e4dbd5dabc64b2f P 2366aa34ca1f4894aa5734518c14fda1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:02.598151 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.594949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.24.11.131:45209)
0928 06:31:02.594988 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.594990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.595369 (+ 379us) server_negotiation.cc:407] Connection header received
0928 06:31:02.595393 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.595394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.595415 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.595465 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:02.595642 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.596370 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.597478 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.597737 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.597792 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.597971 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.597997 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.598037 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.598090 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.598105 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:02.603675 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.600782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55555 (local address 127.24.11.131:45209)
0928 06:31:02.600821 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.600823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.600995 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:31:02.601058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.601066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.601076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.601141 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:02.601274 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.602032 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.602912 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.603214 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.603279 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.603424 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.603455 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.603510 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.603563 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.603582 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:02.603684 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.600729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45209 (local address 127.24.11.131:55555)
0928 06:31:02.600899 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:02.600910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:02.601006 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:02.601146 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:02.601147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:02.601149 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:02.601206 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.601207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.602113 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.602114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.602843 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:02.602845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.603290 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.603291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:02.603343 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.603355 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:02.603521 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:02.603522 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:02.603552 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:31:02.603584 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 06:31:02.606144 26040 raft_consensus.cc:1261] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Refusing update from remote peer 567b08db08614f7ca15efc57b9f947f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.606158 25900 raft_consensus.cc:1261] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Refusing update from remote peer 567b08db08614f7ca15efc57b9f947f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:02.606331 26280 consensus_queue.cc:1035] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c5d13d7b304155a45ca63725dfc2d5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.606410 26255 consensus_queue.cc:1035] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2366aa34ca1f4894aa5734518c14fda1" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:02.618110 25740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:02.618207 25740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:02.618817 25740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff5a0b7d8a5d418b8bfe19919a58cf28: Sending DeleteTablet for 3 replicas of tablet f8a411cff909456cb24e2fde8124a7f2
I20260928 06:31:02.619084 25880 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a411cff909456cb24e2fde8124a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57590
I20260928 06:31:02.619131 26159 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a411cff909456cb24e2fde8124a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56306
I20260928 06:31:02.619176 26020 tablet_service.cc:1514] Processing DeleteTablet for tablet f8a411cff909456cb24e2fde8124a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:02 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:51232
I20260928 06:31:02.619355 26319 tablet_replica.cc:330] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: stopping tablet replica
I20260928 06:31:02.619451 26319 raft_consensus.cc:2229] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:02.619437 26318 tablet_replica.cc:330] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: stopping tablet replica
I20260928 06:31:02.619514 26319 raft_consensus.cc:2258] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.619745 26319 ts_tablet_manager.cc:1905] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.619709 26318 raft_consensus.cc:2229] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:02.619880 26318 raft_consensus.cc:2258] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.620116 26317 tablet_replica.cc:330] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: stopping tablet replica
I20260928 06:31:02.620208 26318 ts_tablet_manager.cc:1905] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.620216 26317 raft_consensus.cc:2229] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:02.620257 26317 raft_consensus.cc:2258] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:02.620675 26317 ts_tablet_manager.cc:1905] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:02.621579 26319 ts_tablet_manager.cc:1918] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:02.621673 26319 log.cc:1196] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/f8a411cff909456cb24e2fde8124a7f2
I20260928 06:31:02.621945 26319 ts_tablet_manager.cc:1939] T f8a411cff909456cb24e2fde8124a7f2 P 2366aa34ca1f4894aa5734518c14fda1: Deleting consensus metadata
I20260928 06:31:02.622046 26318 ts_tablet_manager.cc:1918] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:02.622156 26318 log.cc:1196] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/f8a411cff909456cb24e2fde8124a7f2
I20260928 06:31:02.622301 25726 catalog_manager.cc:4841] TS 2366aa34ca1f4894aa5734518c14fda1 (127.24.11.130:39529): tablet f8a411cff909456cb24e2fde8124a7f2 (table test-table [id=9161be98e44b4f5b838652b640b9fc28]) successfully deleted
I20260928 06:31:02.622469 26318 ts_tablet_manager.cc:1939] T f8a411cff909456cb24e2fde8124a7f2 P 567b08db08614f7ca15efc57b9f947f4: Deleting consensus metadata
I20260928 06:31:02.622761 25728 catalog_manager.cc:4841] TS 567b08db08614f7ca15efc57b9f947f4 (127.24.11.131:45209): tablet f8a411cff909456cb24e2fde8124a7f2 (table test-table [id=9161be98e44b4f5b838652b640b9fc28]) successfully deleted
I20260928 06:31:02.622920 26317 ts_tablet_manager.cc:1918] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:02.622982 26317 log.cc:1196] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/f8a411cff909456cb24e2fde8124a7f2
I20260928 06:31:02.623196 26317 ts_tablet_manager.cc:1939] T f8a411cff909456cb24e2fde8124a7f2 P 33c5d13d7b304155a45ca63725dfc2d5: Deleting consensus metadata
I20260928 06:31:02.623588 25729 catalog_manager.cc:4841] TS 33c5d13d7b304155a45ca63725dfc2d5 (127.24.11.129:42355): tablet f8a411cff909456cb24e2fde8124a7f2 (table test-table [id=9161be98e44b4f5b838652b640b9fc28]) successfully deleted
Sep 28 06:31:02 dist-test-slave-38pg krb5kdc[25697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:31:02.632731 26246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.620026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.24.11.129:42355)
0928 06:31:02.620064 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.620066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.623463 (+ 3397us) server_negotiation.cc:407] Connection header received
0928 06:31:02.623575 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.623576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.623586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.623650 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:02.624252 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.625004 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.625762 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.626089 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.626162 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.629811 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:02.629821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:02.629821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:02.629829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:02.631528 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.631896 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.631901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.631902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.631972 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.632177 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.632178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.632178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.632320 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:02.632486 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.632534 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.632610 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:31:02.632987 25940 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36966
I20260928 06:31:02.639849 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:02.633967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.24.11.190:44223)
0928 06:31:02.634016 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:02.634018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:02.634027 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:02.634097 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:02.634098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:02.634108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:02.634162 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:02.634640 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.635186 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.635863 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:02.636082 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:02.636156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:02.637305 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:02.637306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:02.637306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:02.637314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:02.638737 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.639083 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.639084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.639084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.639140 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:02.639337 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:02.639338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:02.639338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:02.639463 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:02.639566 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:02.639666 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:31:02.639750 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 06:31:02.640105 25740 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52292
I20260928 06:31:02.640722 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25814
I20260928 06:31:02.646628 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25955
I20260928 06:31:02.651685 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 26095
I20260928 06:31:02.657064 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 25710
2026-09-28T06:31:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3247 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:02 dist-test-slave-38pg krb5kdc[26326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:02 dist-test-slave-38pg krb5kdc[26326](info): set up 2 sockets
Sep 28 06:31:02 dist-test-slave-38pg krb5kdc[26326](info): commencing operation
krb5kdc: starting...
W20260928 06:31:03.706439 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:03 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:03.829030 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37025
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:03.910338 26339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:03.910549 26339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:03.910573 26339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:03.911661 26339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:03.911697 26339 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:31:03.911711 26339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:03.911724 26339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:03.911737 26339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:03.911749 26339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:03.913163 26339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:03.913414 26339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:03.913667 26339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:03.918363 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:03.918401 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:03.918373 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:03.918779 26339 server_base.cc:1034] running on GCE node
I20260928 06:31:03.919055 26339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:03.919390 26339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:03.920548 26339 hybrid_clock.cc:648] HybridClock initialized: now 1790577063920520 us; error 45 us; skew 500 ppm
Sep 28 06:31:03 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:03.923116 26339 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:03.923507 26339 webserver.cc:460] Webserver started at http://127.24.11.190:46111/ using document root <none> and password file <none>
I20260928 06:31:03.923708 26339 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:03.923750 26339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:03.923867 26339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:03.924894 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "6bf86cf5162e476194f820d59582f78d"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-38pg"
server_key: "51a38875a3a54d80841c8c9df344a7b6"
server_key_iv: "8db57fe364b64b9f6c64a66a5421e63f"
server_key_version: "encryptionkey@0"
I20260928 06:31:03.925192 26339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "6bf86cf5162e476194f820d59582f78d"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-38pg"
server_key: "51a38875a3a54d80841c8c9df344a7b6"
server_key_iv: "8db57fe364b64b9f6c64a66a5421e63f"
server_key_version: "encryptionkey@0"
I20260928 06:31:03.926578 26339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:03.927263 26354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:03.927430 26339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:03.927538 26339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "6bf86cf5162e476194f820d59582f78d"
format_stamp: "Formatted at 2026-09-28 06:31:03 on dist-test-slave-38pg"
server_key: "51a38875a3a54d80841c8c9df344a7b6"
server_key_iv: "8db57fe364b64b9f6c64a66a5421e63f"
server_key_version: "encryptionkey@0"
I20260928 06:31:03.927613 26339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:03.940748 26339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:03.942759 26339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:03.942929 26339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:03.947688 26339 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:37025
I20260928 06:31:03.947772 26416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:37025 every 8 connection(s)
I20260928 06:31:03.948128 26339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:03.948825 26417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:03.951260 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Bootstrap starting.
I20260928 06:31:03.951862 26417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:03.952158 26417 log.cc:824] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:03.952843 26417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: No bootstrap required, opened a new log
I20260928 06:31:03.954049 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 26339
I20260928 06:31:03.954139 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:03.954296 24622 external_mini_cluster.cc:1428] Setting key 7b89a25f898f67aaae36a6b7d96e8d9c
I20260928 06:31:03.955580 26417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } }
I20260928 06:31:03.955760 26417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:03.955798 26417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf86cf5162e476194f820d59582f78d, State: Initialized, Role: FOLLOWER
I20260928 06:31:03.955979 26417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } }
I20260928 06:31:03.956059 26417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:03.956084 26417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:03.956143 26417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:03.956810 26417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } }
I20260928 06:31:03.956916 26417 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf86cf5162e476194f820d59582f78d; no voters:
I20260928 06:31:03.957079 26417 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:03.957139 26422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:03.957285 26422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [term 1 LEADER]: Becoming Leader. State: Replica: 6bf86cf5162e476194f820d59582f78d, State: Running, Role: LEADER
I20260928 06:31:03.957317 26417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:03.958770 26422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } }
I20260928 06:31:03.959520 26424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf86cf5162e476194f820d59582f78d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } } }
I20260928 06:31:03.959527 26423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf86cf5162e476194f820d59582f78d. Latest consensus state: current_term: 1 leader_uuid: "6bf86cf5162e476194f820d59582f78d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf86cf5162e476194f820d59582f78d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37025 } } }
I20260928 06:31:03.959630 26424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:03.959633 26423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:03.959964 26436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:03.960493 26436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:31:03 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:03.966024 26436 catalog_manager.cc:1347] Generated new cluster ID: 40b84cc4f7f94feabc541710fbaf5b03
I20260928 06:31:03.966120 26436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:03.966545 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:03.954891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.24.11.190:37025)
0928 06:31:03.955113 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:03.955127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:03.955169 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:31:03.955537 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:03.955545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:03.955610 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:03.955772 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:03.956523 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:03.957875 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:03.958830 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:03.959403 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:03.959511 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:03.962518 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:03.962522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:03.962523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:03.962535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:03.964561 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:03.964980 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:03.964983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:03.964984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:03.965109 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:03.965349 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:03.965350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:03.965350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:03.965702 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:03.965867 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:03.966269 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 06:31:03.966392 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260928 06:31:03.979390 26436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:03.980005 26436 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:03.988766 26436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Generated new TSK 0
I20260928 06:31:03.988986 26436 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:04.022222 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:37025
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:04.105782 26443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:04.106006 26443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:04.106033 26443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:04.107257 26443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:04.107306 26443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:04.107327 26443 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:31:04.107378 26443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:04.107405 26443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:04.108906 26443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:04.109153 26443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:04.109401 26443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:04.114985 26449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:04.115016 26448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:04.114990 26451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:04.115202 26443 server_base.cc:1034] running on GCE node
I20260928 06:31:04.115500 26443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:04.115760 26443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:04.116930 26443 hybrid_clock.cc:648] HybridClock initialized: now 1790577064116920 us; error 30 us; skew 500 ppm
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:04.119580 26443 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:04.120042 26443 webserver.cc:460] Webserver started at http://127.24.11.129:35483/ using document root <none> and password file <none>
I20260928 06:31:04.120267 26443 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:04.120342 26443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:04.120500 26443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:04.121740 26443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "fbdaa39ff9fc3245d1f0eec7513052b3"
server_key_iv: "e740e2ce795e4f47808ce3c27ef47305"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.122162 26443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "fbdaa39ff9fc3245d1f0eec7513052b3"
server_key_iv: "e740e2ce795e4f47808ce3c27ef47305"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.123553 26443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:04.124362 26458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.124536 26443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:04.124637 26443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "fbdaa39ff9fc3245d1f0eec7513052b3"
server_key_iv: "e740e2ce795e4f47808ce3c27ef47305"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.124720 26443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:04.151702 26443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:04.153924 26443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:04.154093 26443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:04.154630 26443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:04.154709 26443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.154791 26443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:04.154824 26443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:04.162132 26443 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:33535
I20260928 06:31:04.162302 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:33535 every 8 connection(s)
I20260928 06:31:04.162549 26443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:04.165445 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.155227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60707 (local address 127.24.11.190:37025)
0928 06:31:04.155299 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.155303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.155525 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:31:04.156074 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.156076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.156089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.156170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.157163 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.157999 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.159721 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.160018 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.160102 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.162611 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.162612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.162613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.162621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.164028 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.164441 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.164442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.164443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.164503 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.164762 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.164763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.164763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.164880 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.164951 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.165236 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.165351 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:04.165608 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.155216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.129:60707)
0928 06:31:04.155400 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.155435 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.155961 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.156297 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.156302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.156719 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.157052 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.157059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.158100 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.158101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.159617 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.159621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.160122 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.160123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.160287 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.160876 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.160885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.162510 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.164128 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.164134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.164136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.164362 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.164571 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.164572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.164573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.164688 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.164953 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.164960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.165100 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.165203 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.165364 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260928 06:31:04.166235 26574 heartbeater.cc:347] Connected to a master server at 127.24.11.190:37025
I20260928 06:31:04.166347 26574 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:04.166535 26574 heartbeater.cc:511] Master 127.24.11.190:37025 requested a full tablet report, sending...
I20260928 06:31:04.167076 26370 ts_manager.cc:194] Registered new tserver with Master: 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535)
I20260928 06:31:04.167268 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 26443
I20260928 06:31:04.167392 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:04.167603 24622 external_mini_cluster.cc:1428] Setting key d1f089b5d3d6186ffbdac4ed7b1a7899
I20260928 06:31:04.168002 26370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:60707
I20260928 06:31:04.173631 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.167508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58153 (local address 127.24.11.190:37025)
0928 06:31:04.167542 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.167544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.167551 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:04.167611 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.167612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.167620 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.167676 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.168078 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.168668 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.169740 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.169923 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.169980 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.171093 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.171095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.171095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.171101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.172403 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.172778 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.172779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.172780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.172842 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.173068 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.173069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.173070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.173183 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.173261 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.173440 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.173513 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:04.173640 26580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.167357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.129:58153)
0928 06:31:04.167489 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.167502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.167559 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.167694 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.167695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.167925 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.167983 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.167985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.168766 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.168767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.169643 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.169645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.170005 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.170006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.170093 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.170535 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.170541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.171006 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.172512 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.172513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.172513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.172690 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.172911 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.172912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.172912 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.173001 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.173253 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.173254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.173316 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.173403 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.173513 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:04.194377 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:37025
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:04.275727 26583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:04.275938 26583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:04.275962 26583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:04.277046 26583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:04.277082 26583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:04.277102 26583 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:31:04.277114 26583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:04.277132 26583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:04.278643 26583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:04.278887 26583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:04.279063 26583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:04.284202 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:04.284351 26583 server_base.cc:1034] running on GCE node
W20260928 06:31:04.284261 26588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:04.284209 26589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:04.284643 26583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:04.284930 26583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:04.286110 26583 hybrid_clock.cc:648] HybridClock initialized: now 1790577064286087 us; error 71 us; skew 500 ppm
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:04.288846 26583 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:04.289306 26583 webserver.cc:460] Webserver started at http://127.24.11.130:36625/ using document root <none> and password file <none>
I20260928 06:31:04.289616 26583 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:04.289692 26583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:04.289852 26583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:04.290941 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "27a90dc1a322482f92fbed6c21e88989"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "132b7ca84472ac6c6d0bd9ce5ce412f5"
server_key_iv: "9503f3257b64ba5366263f2218be80bb"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.291318 26583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "27a90dc1a322482f92fbed6c21e88989"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "132b7ca84472ac6c6d0bd9ce5ce412f5"
server_key_iv: "9503f3257b64ba5366263f2218be80bb"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.292651 26583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:04.293363 26598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.293569 26583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:04.293678 26583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "27a90dc1a322482f92fbed6c21e88989"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "132b7ca84472ac6c6d0bd9ce5ce412f5"
server_key_iv: "9503f3257b64ba5366263f2218be80bb"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.293762 26583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:04.302543 26583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:04.304428 26583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:04.304593 26583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:04.305105 26583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:04.305172 26583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.305233 26583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:04.305261 26583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.312209 26583 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:34629
I20260928 06:31:04.312292 26713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:34629 every 8 connection(s)
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:04.312644 26583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:04.315740 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.305684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:50229 (local address 127.24.11.190:37025)
0928 06:31:04.305758 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.305761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.305997 (+ 236us) server_negotiation.cc:407] Connection header received
0928 06:31:04.306480 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.306482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.306494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.306579 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.307404 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.308255 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.309830 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.310165 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.310239 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.313165 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.313167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.313167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.313174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.314408 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.314812 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.314813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.314813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.314863 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.315146 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.315148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.315148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.315317 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.315384 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.315611 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.315676 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:04.315892 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.305691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.130:50229)
0928 06:31:04.305865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.305905 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.306384 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.306656 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.306659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.307016 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.307299 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.307305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.308364 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.308366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.309736 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.309738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.310259 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.310260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.310420 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.311132 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.311151 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.313074 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.314507 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.314512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.314514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.314730 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.314919 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.314919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.314920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.315034 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.315385 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.315389 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.315500 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.315582 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.315698 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260928 06:31:04.316438 26714 heartbeater.cc:347] Connected to a master server at 127.24.11.190:37025
I20260928 06:31:04.316542 26714 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:04.316720 26714 heartbeater.cc:511] Master 127.24.11.190:37025 requested a full tablet report, sending...
I20260928 06:31:04.317173 26371 ts_manager.cc:194] Registered new tserver with Master: 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629)
I20260928 06:31:04.317692 26371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:50229
I20260928 06:31:04.319212 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 26583
I20260928 06:31:04.319331 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:04.319527 24622 external_mini_cluster.cc:1428] Setting key 390156826e5886464721f3e476ce38df
I20260928 06:31:04.324631 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.317593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.130:60981)
0928 06:31:04.317718 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.317728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.317810 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.317967 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.317968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.318158 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.318217 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.318218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.318896 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.318896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.319458 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.319460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.319896 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.319897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.319960 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.321452 (+ 1492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.321456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.321815 (+ 359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.323603 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.323604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.323604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.323764 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.324004 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.324005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.324005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.324056 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.324370 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.324370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.324419 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.324475 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.324555 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260928 06:31:04.324836 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.317630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:60981 (local address 127.24.11.190:37025)
0928 06:31:04.317665 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.317667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.317796 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:31:04.317892 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.317893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.317905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.317959 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.318305 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.318811 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.319539 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.319792 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.319884 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.321902 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.321903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.321904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.321912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.323521 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.323842 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.323844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.323844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.323928 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.324107 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.324107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.324108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.324308 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.324379 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.324519 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.324601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:04.345600 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:37025
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:04.427345 26723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:04.427556 26723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:04.427582 26723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:04.428742 26723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:04.428787 26723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:04.428808 26723 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:31:04.428820 26723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:04.428838 26723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:04.430342 26723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:37025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:04.430531 26723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:04.430708 26723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:04.435882 26723 server_base.cc:1034] running on GCE node
W20260928 06:31:04.435932 26729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:04.435869 26728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:04.435853 26731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:04.436268 26723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:04.436472 26723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:04.437587 26723 hybrid_clock.cc:648] HybridClock initialized: now 1790577064437576 us; error 31 us; skew 500 ppm
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:04.440047 26723 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:04.440521 26723 webserver.cc:460] Webserver started at http://127.24.11.131:41523/ using document root <none> and password file <none>
I20260928 06:31:04.440764 26723 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:04.440832 26723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:04.440995 26723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:04.442128 26723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "d731b2fc5d284af28b8a47a53844e393"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "270ba2d1458efb7a38fa400ad16c5ffb"
server_key_iv: "8189bfe277a2e39eba2bff6a501f036b"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.442498 26723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "d731b2fc5d284af28b8a47a53844e393"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "270ba2d1458efb7a38fa400ad16c5ffb"
server_key_iv: "8189bfe277a2e39eba2bff6a501f036b"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.443904 26723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:04.444669 26738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.444872 26723 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:04.444957 26723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "d731b2fc5d284af28b8a47a53844e393"
format_stamp: "Formatted at 2026-09-28 06:31:04 on dist-test-slave-38pg"
server_key: "270ba2d1458efb7a38fa400ad16c5ffb"
server_key_iv: "8189bfe277a2e39eba2bff6a501f036b"
server_key_version: "encryptionkey@0"
I20260928 06:31:04.445009 26723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:04.455603 26723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:04.458005 26723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:04.458148 26723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:04.458794 26723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:04.458835 26723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.458858 26723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:04.458869 26723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:04.465837 26723 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:35355
I20260928 06:31:04.465909 26853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:35355 every 8 connection(s)
I20260928 06:31:04.466255 26723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:04.469089 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.459262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:48319 (local address 127.24.11.190:37025)
0928 06:31:04.459334 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.459337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.459509 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:31:04.460075 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.460076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.460091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.460175 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.461019 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.461897 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.462896 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.463239 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.463315 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.465838 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.465840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.465840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.465849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.467681 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.468100 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.468101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.468101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.468160 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.468430 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.468431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.468431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.468556 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.468636 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.468908 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.468995 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:04.469286 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.459232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.131:48319)
0928 06:31:04.459380 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.459420 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.459896 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.460263 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.460269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.460618 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.460897 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.460904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.461993 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.461994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.462816 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.462819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.463343 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.463344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.463483 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.464004 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.464036 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.465742 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.467782 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.467788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.467790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.468022 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.468238 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.468239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.468239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.468353 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.468626 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.468632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.468773 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.468885 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.469050 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260928 06:31:04.469821 26854 heartbeater.cc:347] Connected to a master server at 127.24.11.190:37025
I20260928 06:31:04.469923 26854 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:04.470093 26854 heartbeater.cc:511] Master 127.24.11.190:37025 requested a full tablet report, sending...
I20260928 06:31:04.470497 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 26723
I20260928 06:31:04.470520 26370 ts_manager.cc:194] Registered new tserver with Master: d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:04.470598 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:04.470806 24622 external_mini_cluster.cc:1428] Setting key 0d2188fb6fa4d15012d06a20fb4675d1
I20260928 06:31:04.471027 26370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:48319
I20260928 06:31:04.472038 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:04.476970 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.470872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52667 (local address 127.24.11.190:37025)
0928 06:31:04.471158 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.471159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.471167 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:04.471186 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.471187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.471197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.471248 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.471637 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.472252 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.472948 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.473154 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.473226 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.474209 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.474211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.474211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.474218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.475783 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.476153 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.476154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.476155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.476216 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.476422 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.476423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.476424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.476565 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.476638 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.476817 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.476893 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260928 06:31:04.477988 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.470805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.24.11.131:52667)
0928 06:31:04.470935 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.470944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.470986 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.471298 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.471299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.471476 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.471543 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.471545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.472341 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.472342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.472858 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.472860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.473235 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.473236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.473312 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.473813 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.473817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.474100 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.475887 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.475888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.475889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.476072 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.476285 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.476286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.476286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.476356 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.476635 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.476636 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.476694 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.476772 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.476923 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:04.495478 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.488592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.24.11.190:37025)
0928 06:31:04.488667 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.488669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.488676 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:04.488698 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.488699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.488707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.488781 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.489489 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.490052 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.490749 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.490931 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.490981 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.493193 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.493194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.493194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.493201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.494575 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.494886 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.494887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.494887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.494938 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.495101 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.495101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.495102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.495198 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.495273 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.495356 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.495413 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:04.497088 26370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:04.498824 26370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:04.505613 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.24.11.130:34629)
0928 06:31:04.501619 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.501624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.502106 (+ 482us) server_negotiation.cc:407] Connection header received
0928 06:31:04.502232 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.502235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.502252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.502306 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:04.502495 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503167 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.504448 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505152 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505232 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505292 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.505452 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.505491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260928 06:31:04.505609 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:34629 (local address 127.0.0.1:47452)
0928 06:31:04.502019 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.502029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.502135 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.502393 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.502393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.502395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:04.502426 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.502427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503453 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.503453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.504360 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.504362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505247 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.505331 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505345 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.505418 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.505476 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":62,"threads_started":1}
I20260928 06:31:04.505823 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.24.11.129:33535)
0928 06:31:04.501693 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.501696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.501704 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:04.501740 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.501742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.501754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.501850 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:04.502011 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503158 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.504694 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505183 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505275 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505332 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.505649 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.505693 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":93,"threads_started":1}
I20260928 06:31:04.505980 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.24.11.131:35355)
0928 06:31:04.501620 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.501625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.501720 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:31:04.501815 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.501817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.501832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.501921 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:04.502158 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503188 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.504681 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505372 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505456 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505504 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.505871 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.505902 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260928 06:31:04.505818 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.0.0.1:54290)
0928 06:31:04.501641 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.501651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.501732 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.501917 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.501918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.501920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:04.501975 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.501977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503287 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.503288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.504593 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.504595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505542 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505572 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.505648 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505652 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.505709 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.505752 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":80,"thread_start_us":102,"threads_started":1}
I20260928 06:31:04.505607 26872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.501107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33535 (local address 127.0.0.1:57406)
0928 06:31:04.501291 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.501345 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.501412 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.501855 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.501858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.501861 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:04.501939 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.501942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.503264 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.503265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.504592 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.504595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.505362 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.505363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.505422 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.505425 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.505470 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.505499 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":55,"thread_start_us":48,"threads_started":1}
I20260928 06:31:04.506953 26788 tablet_service.cc:1467] Processing CreateTablet for tablet 742bb6a172324ae9ab69ed00f142806c (DEFAULT_TABLE table=test-table [id=a4f8c054f3524612b31d34e53a12b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:04.507000 26508 tablet_service.cc:1467] Processing CreateTablet for tablet 742bb6a172324ae9ab69ed00f142806c (DEFAULT_TABLE table=test-table [id=a4f8c054f3524612b31d34e53a12b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:04.507323 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742bb6a172324ae9ab69ed00f142806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.507324 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742bb6a172324ae9ab69ed00f142806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.510035 26648 tablet_service.cc:1467] Processing CreateTablet for tablet 742bb6a172324ae9ab69ed00f142806c (DEFAULT_TABLE table=test-table [id=a4f8c054f3524612b31d34e53a12b4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:04.510366 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742bb6a172324ae9ab69ed00f142806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.510901 26877 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Bootstrap starting.
I20260928 06:31:04.511191 26878 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Bootstrap starting.
I20260928 06:31:04.511721 26878 tablet_bootstrap.cc:654] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.511979 26877 tablet_bootstrap.cc:654] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.512056 26878 log.cc:824] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:04.512301 26877 log.cc:824] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:04.512799 26878 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: No bootstrap required, opened a new log
I20260928 06:31:04.512871 26878 ts_tablet_manager.cc:1397] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:04.513051 26877 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: No bootstrap required, opened a new log
I20260928 06:31:04.513118 26877 ts_tablet_manager.cc:1397] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:04.514006 26881 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Bootstrap starting.
I20260928 06:31:04.514605 26881 tablet_bootstrap.cc:654] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.514922 26881 log.cc:824] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:04.514930 26878 raft_consensus.cc:348] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.515107 26878 raft_consensus.cc:374] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.515055 26877 raft_consensus.cc:348] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.515153 26878 raft_consensus.cc:729] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.515189 26877 raft_consensus.cc:374] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.515219 26877 raft_consensus.cc:729] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.515312 26877 consensus_queue.cc:260] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.515308 26878 consensus_queue.cc:260] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.515578 26877 ts_tablet_manager.cc:1428] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 06:31:04.515578 26878 ts_tablet_manager.cc:1428] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:04.515693 26854 heartbeater.cc:503] Master 127.24.11.190:37025 was elected leader, sending a full tablet report...
I20260928 06:31:04.515826 26881 tablet_bootstrap.cc:492] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: No bootstrap required, opened a new log
I20260928 06:31:04.515888 26574 heartbeater.cc:503] Master 127.24.11.190:37025 was elected leader, sending a full tablet report...
I20260928 06:31:04.515921 26881 ts_tablet_manager.cc:1397] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:04.517584 26881 raft_consensus.cc:348] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.517755 26881 raft_consensus.cc:374] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.517798 26881 raft_consensus.cc:729] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.517910 26881 consensus_queue.cc:260] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.518200 26881 ts_tablet_manager.cc:1428] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:04.518220 26714 heartbeater.cc:503] Master 127.24.11.190:37025 was elected leader, sending a full tablet report...
W20260928 06:31:04.563251 26715 tablet.cc:2369] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:04.663352 26575 tablet.cc:2369] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:04.715102 26884 raft_consensus.cc:482] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:04.715236 26884 raft_consensus.cc:504] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.715687 26884 leader_election.cc:288] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355), 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629)
W20260928 06:31:04.716832 26855 tablet.cc:2369] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:04.718957 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.716301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:34743 (local address 127.24.11.130:34629)
0928 06:31:04.716352 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.716355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.716363 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:04.716386 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.716387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.716397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.716500 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:04.716629 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.717231 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718245 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.718605 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718660 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.718666 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.718830 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.718888 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:31:04.718973 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.716079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:37287 (local address 127.24.11.131:35355)
0928 06:31:04.716142 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.716144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.716199 (+ 55us) server_negotiation.cc:407] Connection header received
0928 06:31:04.716230 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.716231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.716241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.716298 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:04.716518 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.717223 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718243 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.718620 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718710 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.718718 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.718877 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.718916 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:04.718962 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.716142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:34629 (local address 127.24.11.129:34743)
0928 06:31:04.716246 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.716257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.716305 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.716484 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.716484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.716486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:04.716553 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.716555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.717336 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.717337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.718150 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.718152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.718683 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.718756 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.718760 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.718809 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.718868 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":10,"thread_start_us":50,"threads_started":1}
I20260928 06:31:04.718976 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.715957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.24.11.129:37287)
0928 06:31:04.716071 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.716084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.716139 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.716332 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.716333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.716336 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:04.716423 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.716425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.717304 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.717306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.718147 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.718150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.718726 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.718727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.718782 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.718785 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.718856 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.718896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260928 06:31:04.719417 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742bb6a172324ae9ab69ed00f142806c" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989" is_pre_election: true
I20260928 06:31:04.719411 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742bb6a172324ae9ab69ed00f142806c" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393" is_pre_election: true
I20260928 06:31:04.719578 26802 raft_consensus.cc:2454] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 0.
I20260928 06:31:04.719574 26668 raft_consensus.cc:2454] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 0.
I20260928 06:31:04.719799 26460 leader_election.cc:302] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:04.719919 26884 raft_consensus.cc:2781] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:04.720005 26884 raft_consensus.cc:482] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:04.720041 26884 raft_consensus.cc:3037] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:04.720733 26884 raft_consensus.cc:504] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.720871 26884 leader_election.cc:288] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 election: Requested vote from peers d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355), 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629)
I20260928 06:31:04.721041 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742bb6a172324ae9ab69ed00f142806c" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393"
I20260928 06:31:04.721069 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742bb6a172324ae9ab69ed00f142806c" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989"
I20260928 06:31:04.721117 26802 raft_consensus.cc:3037] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:04.721133 26668 raft_consensus.cc:3037] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:04.721887 26668 raft_consensus.cc:2454] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 1.
I20260928 06:31:04.721909 26802 raft_consensus.cc:2454] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 1.
I20260928 06:31:04.722052 26460 leader_election.cc:302] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, 27a90dc1a322482f92fbed6c21e88989; no voters:
I20260928 06:31:04.722182 26884 raft_consensus.cc:2781] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:04.722323 26884 raft_consensus.cc:686] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 LEADER]: Becoming Leader. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Running, Role: LEADER
I20260928 06:31:04.722432 26884 consensus_queue.cc:237] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } }
I20260928 06:31:04.723310 26371 catalog_manager.cc:5495] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129). New cstate: current_term: 1 leader_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:31:04 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:04.762697 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.754340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.24.11.190:37025)
0928 06:31:04.754393 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:04.754395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:04.754573 (+ 178us) server_negotiation.cc:407] Connection header received
0928 06:31:04.754655 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:04.754656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:04.754667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:04.754753 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:04.755242 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.755840 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.756681 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.756942 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.757291 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.759715 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:04.759716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:04.759716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:04.759723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:04.761488 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.761901 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.761902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.761903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.761954 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:04.762153 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:04.762154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:04.762155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:04.762269 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:04.762352 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:04.762506 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:31:04.762589 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:31:04.762768 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:04.754273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37025 (local address 127.0.0.1:33206)
0928 06:31:04.754499 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:04.754509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:04.754594 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:04.754758 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:04.754759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:04.755108 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:04.755151 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.755154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.756033 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.756034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.756576 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:04.756577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:04.757303 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:04.757304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:04.757380 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:04.758113 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:04.758117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:04.759603 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:04.761594 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.761602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.761605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.761809 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.762026 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:04.762027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:04.762028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:04.762088 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:04.762355 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:04.762363 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:04.762412 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:31:04.762484 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:31:04.762646 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":64,"thread_start_us":127,"threads_started":1}
I20260928 06:31:04.764895 26371 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:33206:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:31:04.765126 26371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:04.766682 26508 tablet_service.cc:1467] Processing CreateTablet for tablet b70121a1f8c945b6b64cb39bd09744d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b749174a8ed4692982a546ced03f29c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:04.766752 26788 tablet_service.cc:1467] Processing CreateTablet for tablet b70121a1f8c945b6b64cb39bd09744d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b749174a8ed4692982a546ced03f29c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:04.766722 26648 tablet_service.cc:1467] Processing CreateTablet for tablet b70121a1f8c945b6b64cb39bd09744d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b749174a8ed4692982a546ced03f29c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:04.766885 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70121a1f8c945b6b64cb39bd09744d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.766898 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70121a1f8c945b6b64cb39bd09744d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.766886 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70121a1f8c945b6b64cb39bd09744d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:04.768211 26878 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: Bootstrap starting.
I20260928 06:31:04.768370 26881 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: Bootstrap starting.
I20260928 06:31:04.768393 26877 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: Bootstrap starting.
I20260928 06:31:04.768671 26878 tablet_bootstrap.cc:654] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.768939 26877 tablet_bootstrap.cc:654] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.769088 26881 tablet_bootstrap.cc:654] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:04.769699 26877 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: No bootstrap required, opened a new log
I20260928 06:31:04.769757 26877 ts_tablet_manager.cc:1397] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:04.769908 26881 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: No bootstrap required, opened a new log
I20260928 06:31:04.769907 26877 raft_consensus.cc:348] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.769973 26881 ts_tablet_manager.cc:1397] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:04.769976 26877 raft_consensus.cc:374] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.770016 26877 raft_consensus.cc:729] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.770081 26877 consensus_queue.cc:260] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.770169 26877 ts_tablet_manager.cc:1428] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:04.770144 26881 raft_consensus.cc:348] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.770231 26881 raft_consensus.cc:374] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.770262 26881 raft_consensus.cc:729] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.770313 26881 consensus_queue.cc:260] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.770323 26883 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:04.770409 26883 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:04.770406 26881 ts_tablet_manager.cc:1428] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:04.770463 26885 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:04.770527 26885 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:04.770946 26878 tablet_bootstrap.cc:492] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: No bootstrap required, opened a new log
I20260928 06:31:04.770996 26878 ts_tablet_manager.cc:1397] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:31:04.771307 26878 raft_consensus.cc:348] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.771392 26878 raft_consensus.cc:374] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:04.771421 26878 raft_consensus.cc:729] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Initialized, Role: FOLLOWER
I20260928 06:31:04.771459 26878 consensus_queue.cc:260] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:04.771556 26878 ts_tablet_manager.cc:1428] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:04.771600 26884 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:04.771697 26884 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:05.042949 26883 raft_consensus.cc:482] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:05.043052 26883 raft_consensus.cc:504] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.043378 26883 leader_election.cc:288] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629)
I20260928 06:31:05.046275 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.043636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:59909 (local address 127.24.11.130:34629)
0928 06:31:05.043684 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.043686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.043787 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:31:05.043858 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.043859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.043874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.043942 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:05.044101 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.044587 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.045564 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.045921 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.046015 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.046024 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.046176 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.046211 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:31:05.046286 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.043573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:34629 (local address 127.24.11.131:59909)
0928 06:31:05.043701 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.043711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.043798 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.043948 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.043948 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.043950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:05.044029 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.044031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.044672 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.044673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.045450 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.045452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.046013 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.046014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.046092 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.046095 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.046157 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.046199 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260928 06:31:05.046331 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.043627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:60517 (local address 127.24.11.129:33535)
0928 06:31:05.043730 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.043732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.043777 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:31:05.043872 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.043873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.043883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.043974 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:05.044133 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.044613 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.045568 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.045969 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.046057 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.046065 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.046212 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.046249 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260928 06:31:05.046420 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.043576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33535 (local address 127.24.11.131:60517)
0928 06:31:05.043711 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.043722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.043792 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.043976 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.043977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.043979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:05.044059 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.044060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.044700 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.044701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.045457 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.045459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.046066 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.046067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.046139 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.046142 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.046208 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.046248 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260928 06:31:05.046617 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70121a1f8c945b6b64cb39bd09744d5" candidate_uuid: "d731b2fc5d284af28b8a47a53844e393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989" is_pre_election: true
I20260928 06:31:05.046721 26668 raft_consensus.cc:2454] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731b2fc5d284af28b8a47a53844e393 in term 0.
I20260928 06:31:05.046694 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70121a1f8c945b6b64cb39bd09744d5" candidate_uuid: "d731b2fc5d284af28b8a47a53844e393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" is_pre_election: true
I20260928 06:31:05.046804 26523 raft_consensus.cc:2454] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d731b2fc5d284af28b8a47a53844e393 in term 0.
I20260928 06:31:05.046887 26740 leader_election.cc:302] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a90dc1a322482f92fbed6c21e88989, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:05.047029 26883 raft_consensus.cc:2781] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:05.047091 26883 raft_consensus.cc:482] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:05.047124 26883 raft_consensus.cc:3037] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.047830 26883 raft_consensus.cc:504] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.047988 26883 leader_election.cc:288] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [CANDIDATE]: Term 1 election: Requested vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629)
I20260928 06:31:05.048133 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70121a1f8c945b6b64cb39bd09744d5" candidate_uuid: "d731b2fc5d284af28b8a47a53844e393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989"
I20260928 06:31:05.048221 26668 raft_consensus.cc:3037] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.048189 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70121a1f8c945b6b64cb39bd09744d5" candidate_uuid: "d731b2fc5d284af28b8a47a53844e393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
I20260928 06:31:05.048265 26523 raft_consensus.cc:3037] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.048803 26668 raft_consensus.cc:2454] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731b2fc5d284af28b8a47a53844e393 in term 1.
I20260928 06:31:05.049029 26740 leader_election.cc:302] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a90dc1a322482f92fbed6c21e88989, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:05.049048 26523 raft_consensus.cc:2454] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d731b2fc5d284af28b8a47a53844e393 in term 1.
I20260928 06:31:05.049117 26883 raft_consensus.cc:2781] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:05.049237 26883 raft_consensus.cc:686] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [term 1 LEADER]: Becoming Leader. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Running, Role: LEADER
I20260928 06:31:05.049329 26883 consensus_queue.cc:237] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.049803 26902 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: TxnStatusTablet state changed. Reason: New leader d731b2fc5d284af28b8a47a53844e393. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.049896 26902 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:05.050066 26904 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:31:05.050201 26904 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:31:05.050166 26370 catalog_manager.cc:5495] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 reported cstate change: term changed from 0 to 1, leader changed from <none> to d731b2fc5d284af28b8a47a53844e393 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:05.120827 26884 consensus_queue.cc:1035] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.124017 26884 consensus_queue.cc:1035] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.126884 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.124039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.24.11.131:35355)
0928 06:31:05.124186 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.124188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.124196 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:05.124224 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.124225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.124244 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.124311 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.124451 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.124933 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.126034 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.126305 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.126358 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.126573 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.126658 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.126716 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.126799 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.126817 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 06:31:05.126928 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.123659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.0.0.1:54292)
0928 06:31:05.123744 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.123756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.123818 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.124320 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.124321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.124322 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:05.124375 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.124378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.125036 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.125038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.125930 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.125934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.126383 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.126385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.126461 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.126478 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.126727 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.126735 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.126795 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.126846 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:31:05.128535 26668 raft_consensus.cc:1261] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Refusing update from remote peer d731b2fc5d284af28b8a47a53844e393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:05.128655 26523 raft_consensus.cc:1261] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Refusing update from remote peer d731b2fc5d284af28b8a47a53844e393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:05.128782 26902 consensus_queue.cc:1035] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.128908 26883 consensus_queue.cc:1035] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.129791 26885 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: TxnStatusTablet state changed. Reason: New leader d731b2fc5d284af28b8a47a53844e393. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.129882 26885 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:05.130067 26884 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: TxnStatusTablet state changed. Reason: New leader d731b2fc5d284af28b8a47a53844e393. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.130184 26884 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:05.131014 26883 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.131131 26883 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:05.131078 26902 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.131182 26902 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P d731b2fc5d284af28b8a47a53844e393: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:05.131233 26907 mvcc.cc:205] Tried to move back new op lower bound from 7334203658764185600 to 7334203658442563584. Current Snapshot: MvccSnapshot[applied={T|T < 7334203658764185600}]
I20260928 06:31:05.131295 26909 mvcc.cc:205] Tried to move back new op lower bound from 7334203658764185600 to 7334203658442563584. Current Snapshot: MvccSnapshot[applied={T|T < 7334203658764185600}]
I20260928 06:31:05.131316 26884 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.131382 26884 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 17140b4cafc847f1a0780e5b4a2f38ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:05.131739 26885 tablet_replica.cc:439] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d731b2fc5d284af28b8a47a53844e393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } } }
I20260928 06:31:05.131858 26885 tablet_replica.cc:442] T b70121a1f8c945b6b64cb39bd09744d5 P 27a90dc1a322482f92fbed6c21e88989: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:05.132181 26911 mvcc.cc:205] Tried to move back new op lower bound from 7334203658764185600 to 7334203658442563584. Current Snapshot: MvccSnapshot[applied={T|T < 7334203658764185600 or (T in {7334203658764185600})}]
I20260928 06:31:05.136718 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.134055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.24.11.129:33535)
0928 06:31:05.134096 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.134098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.134176 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:31:05.134235 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.134236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.134254 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.134341 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.134435 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.135160 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.135915 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.136218 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.136295 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.136402 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.136522 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.136605 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.136622 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.136645 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:05.141898 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.139306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:40917 (local address 127.24.11.131:35355)
0928 06:31:05.139354 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.139357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.139403 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:31:05.139477 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.139477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.139487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.139530 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.139684 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.140218 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.141044 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.141270 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.141369 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.141635 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.141675 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.141740 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.141802 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.141822 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:05.141942 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.139218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.24.11.129:40917)
0928 06:31:05.139350 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.139363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.139420 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.139530 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.139531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.139532 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:05.139601 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.139602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.140291 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.140293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.140955 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.140957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.141370 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.141371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.141445 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.141459 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.141740 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.141743 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.141798 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.141841 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260928 06:31:05.151539 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.148955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:33819 (local address 127.24.11.129:33535)
0928 06:31:05.148996 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.148999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.149121 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:31:05.149167 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.149168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.149178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.149236 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.149365 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.149933 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.150712 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.150996 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.151061 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.151236 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.151272 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.151320 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.151439 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.151459 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:05.151576 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.148893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33535 (local address 127.24.11.131:33819)
0928 06:31:05.149049 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.149061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.149117 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.149239 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.149240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.149242 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:05.149307 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.149309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.150016 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.150017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.150607 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.150609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.151060 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.151061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.151144 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.151160 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.151345 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.151347 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.151412 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.151459 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260928 06:31:05.161604 26371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:05.161720 26371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:05.162586 26371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Sending DeleteTablet for 3 replicas of tablet 742bb6a172324ae9ab69ed00f142806c
I20260928 06:31:05.162873 26788 tablet_service.cc:1514] Processing DeleteTablet for tablet 742bb6a172324ae9ab69ed00f142806c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:05 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54290
I20260928 06:31:05.163117 26648 tablet_service.cc:1514] Processing DeleteTablet for tablet 742bb6a172324ae9ab69ed00f142806c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:05 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:47452
I20260928 06:31:05.163601 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet 742bb6a172324ae9ab69ed00f142806c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:05 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57406
I20260928 06:31:05.164110 26371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:05.164240 26371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:05.165254 26931 tablet_replica.cc:330] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: stopping tablet replica
I20260928 06:31:05.165426 26931 raft_consensus.cc:2229] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:05.165617 26931 raft_consensus.cc:2258] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:05.166198 26929 tablet_replica.cc:330] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: stopping tablet replica
I20260928 06:31:05.166311 26508 tablet_service.cc:1467] Processing CreateTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 (DEFAULT_TABLE table=test-table [id=e510e88c277940c392d81c32f6ded3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:05.166352 26930 tablet_replica.cc:330] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: stopping tablet replica
I20260928 06:31:05.166448 26930 raft_consensus.cc:2229] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:05.166450 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fbf68dc40144c6bb4643f3214b1939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:05.166074 26931 ts_tablet_manager.cc:1905] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:05.167280 26929 raft_consensus.cc:2229] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:05.167351 26929 raft_consensus.cc:2258] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:05.167446 26930 raft_consensus.cc:2258] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:05.167815 26929 ts_tablet_manager.cc:1905] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:05.167845 26648 tablet_service.cc:1467] Processing CreateTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 (DEFAULT_TABLE table=test-table [id=e510e88c277940c392d81c32f6ded3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:05.167945 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fbf68dc40144c6bb4643f3214b1939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:05.169270 26788 tablet_service.cc:1467] Processing CreateTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 (DEFAULT_TABLE table=test-table [id=e510e88c277940c392d81c32f6ded3f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:05.169384 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2fbf68dc40144c6bb4643f3214b1939. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:05.170724 26878 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Bootstrap starting.
I20260928 06:31:05.171015 26931 ts_tablet_manager.cc:1918] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:05.171082 26931 log.cc:1196] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/742bb6a172324ae9ab69ed00f142806c
I20260928 06:31:05.171128 26878 tablet_bootstrap.cc:654] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:05.171360 26931 ts_tablet_manager.cc:1939] T 742bb6a172324ae9ab69ed00f142806c P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting consensus metadata
I20260928 06:31:05.171469 26881 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Bootstrap starting.
I20260928 06:31:05.171836 26881 tablet_bootstrap.cc:654] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:05.172017 26929 ts_tablet_manager.cc:1918] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:05.172077 26929 log.cc:1196] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/742bb6a172324ae9ab69ed00f142806c
I20260928 06:31:05.172166 26877 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Bootstrap starting.
I20260928 06:31:05.172319 26929 ts_tablet_manager.cc:1939] T 742bb6a172324ae9ab69ed00f142806c P d731b2fc5d284af28b8a47a53844e393: Deleting consensus metadata
I20260928 06:31:05.172546 26877 tablet_bootstrap.cc:654] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:05.173081 26355 catalog_manager.cc:4841] TS 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535): tablet 742bb6a172324ae9ab69ed00f142806c (table test-table [id=a4f8c054f3524612b31d34e53a12b4e1]) successfully deleted
I20260928 06:31:05.173128 26877 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: No bootstrap required, opened a new log
I20260928 06:31:05.173169 26877 ts_tablet_manager.cc:1397] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:05.173264 26356 catalog_manager.cc:4841] TS d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355): tablet 742bb6a172324ae9ab69ed00f142806c (table test-table [id=a4f8c054f3524612b31d34e53a12b4e1]) successfully deleted
I20260928 06:31:05.173295 26877 raft_consensus.cc:348] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.173368 26877 raft_consensus.cc:374] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:05.173388 26877 raft_consensus.cc:729] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Initialized, Role: FOLLOWER
I20260928 06:31:05.173446 26877 consensus_queue.cc:260] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.173555 26877 ts_tablet_manager.cc:1428] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:05.174022 26881 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: No bootstrap required, opened a new log
I20260928 06:31:05.174062 26881 ts_tablet_manager.cc:1397] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:31:05.174189 26881 raft_consensus.cc:348] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.174249 26881 raft_consensus.cc:374] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:05.174268 26881 raft_consensus.cc:729] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Initialized, Role: FOLLOWER
I20260928 06:31:05.174307 26881 consensus_queue.cc:260] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.174382 26881 ts_tablet_manager.cc:1428] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.177855 26930 ts_tablet_manager.cc:1905] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:05.177970 26878 tablet_bootstrap.cc:492] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: No bootstrap required, opened a new log
I20260928 06:31:05.178028 26878 ts_tablet_manager.cc:1397] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 06:31:05.178174 26878 raft_consensus.cc:348] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.178246 26878 raft_consensus.cc:374] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:05.178264 26878 raft_consensus.cc:729] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Initialized, Role: FOLLOWER
I20260928 06:31:05.178306 26878 consensus_queue.cc:260] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.178391 26878 ts_tablet_manager.cc:1428] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:05.179661 26930 ts_tablet_manager.cc:1918] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:05.179733 26930 log.cc:1196] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/742bb6a172324ae9ab69ed00f142806c
I20260928 06:31:05.179986 26930 ts_tablet_manager.cc:1939] T 742bb6a172324ae9ab69ed00f142806c P 27a90dc1a322482f92fbed6c21e88989: Deleting consensus metadata
I20260928 06:31:05.180514 26356 catalog_manager.cc:4841] TS 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629): tablet 742bb6a172324ae9ab69ed00f142806c (table test-table [id=a4f8c054f3524612b31d34e53a12b4e1]) successfully deleted
I20260928 06:31:05.319166 26885 raft_consensus.cc:482] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:05.319270 26885 raft_consensus.cc:504] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.319545 26885 leader_election.cc:288] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:05.322597 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.319864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:34029 (local address 127.24.11.131:35355)
0928 06:31:05.319913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.319914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.319946 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:31:05.320050 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.320051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.320061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.320131 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:05.320300 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.320756 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.321910 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.322255 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.322342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.322352 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.322493 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.322529 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:05.322611 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.319734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.24.11.130:34029)
0928 06:31:05.319868 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.319883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.319966 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.320146 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.320147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.320149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:05.320224 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.320226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.320839 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.320840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.321822 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.321824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.322347 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.322348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.322407 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.322411 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.322485 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.322524 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260928 06:31:05.322664 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.319870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:39901 (local address 127.24.11.129:33535)
0928 06:31:05.319915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.319917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.319975 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:31:05.320046 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.320047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.320057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.320132 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:05.320288 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.320759 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.321912 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.322275 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.322371 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.322382 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.322554 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.322593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:05.322715 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.319734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33535 (local address 127.24.11.130:39901)
0928 06:31:05.319900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.319911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.319987 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.320139 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.320140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.320142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:05.320216 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.320217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.320870 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.320871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.321819 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.321822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.322390 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.322392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.322457 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.322476 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.322545 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.322585 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260928 06:31:05.322968 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fbf68dc40144c6bb4643f3214b1939" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" is_pre_election: true
I20260928 06:31:05.323047 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fbf68dc40144c6bb4643f3214b1939" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393" is_pre_election: true
I20260928 06:31:05.323093 26523 raft_consensus.cc:2454] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 0.
I20260928 06:31:05.323133 26802 raft_consensus.cc:2454] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 0.
I20260928 06:31:05.323310 26599 leader_election.cc:302] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, 27a90dc1a322482f92fbed6c21e88989; no voters:
I20260928 06:31:05.323451 26885 raft_consensus.cc:2781] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:05.323491 26885 raft_consensus.cc:482] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:05.323516 26885 raft_consensus.cc:3037] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.324167 26885 raft_consensus.cc:504] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.324317 26885 leader_election.cc:288] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 election: Requested vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:05.324481 26523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fbf68dc40144c6bb4643f3214b1939" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
I20260928 06:31:05.324555 26523 raft_consensus.cc:3037] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.324501 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2fbf68dc40144c6bb4643f3214b1939" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393"
I20260928 06:31:05.324595 26802 raft_consensus.cc:3037] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:05.325165 26523 raft_consensus.cc:2454] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 1.
I20260928 06:31:05.325279 26802 raft_consensus.cc:2454] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 1.
I20260928 06:31:05.325346 26599 leader_election.cc:302] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, 27a90dc1a322482f92fbed6c21e88989; no voters:
I20260928 06:31:05.325433 26885 raft_consensus.cc:2781] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:05.325613 26885 raft_consensus.cc:686] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 1 LEADER]: Becoming Leader. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Running, Role: LEADER
I20260928 06:31:05.325695 26885 consensus_queue.cc:237] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:05.326419 26370 catalog_manager.cc:5495] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "27a90dc1a322482f92fbed6c21e88989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:05.353332 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.350859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.24.11.130:34629)
0928 06:31:05.350897 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.350899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.350907 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:05.351011 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.351011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.351034 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.351122 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.351229 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.351708 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.352460 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.352670 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.352746 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.352915 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.353030 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.353110 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.353173 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.353197 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:31:05.358292 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.356013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:49053 (local address 127.24.11.131:35355)
0928 06:31:05.356072 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.356074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.356206 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:31:05.356244 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.356245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.356253 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.356304 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.356412 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.356885 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.357662 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.357878 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.357935 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.358068 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.358092 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.358125 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.358179 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.358193 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:31:05.358363 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.355949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35355 (local address 127.24.11.130:49053)
0928 06:31:05.356139 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.356149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.356202 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.356313 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.356313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.356315 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:05.356374 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.356375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.356954 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.356955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.357597 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.357599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.357953 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.357953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.357997 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.358009 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.358146 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.358147 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.358179 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.358209 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260928 06:31:05.360745 26523 raft_consensus.cc:1261] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Refusing update from remote peer 27a90dc1a322482f92fbed6c21e88989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:05.360798 26802 raft_consensus.cc:1261] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Refusing update from remote peer 27a90dc1a322482f92fbed6c21e88989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:05.361001 26885 consensus_queue.cc:1035] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.361150 26937 consensus_queue.cc:1035] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:05.367974 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.365809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42247 (local address 127.24.11.130:34629)
0928 06:31:05.365847 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:05.365849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:05.365874 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:05.365944 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:05.365944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:05.365955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:05.366009 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:05.366125 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.366658 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.367341 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.367521 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.367599 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.367754 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.367786 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.367833 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:05.367890 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:31:05.367907 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:05.368008 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:05.365759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:34629 (local address 127.24.11.131:42247)
0928 06:31:05.365823 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:05.365836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:05.365892 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:05.366013 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:05.366013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:05.366014 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:05.366057 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.366058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.366716 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.366717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.367261 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:05.367263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:05.367600 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:05.367601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:05.367660 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:05.367665 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:05.367834 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:05.367835 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:05.367886 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:05.367935 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:31:05.368351 26911 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:31:06.887414 26371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:06.887524 26371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33196:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:06.888338 26371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Sending DeleteTablet for 3 replicas of tablet c2fbf68dc40144c6bb4643f3214b1939
I20260928 06:31:06.888602 26648 tablet_service.cc:1514] Processing DeleteTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:47452
I20260928 06:31:06.888621 26788 tablet_service.cc:1514] Processing DeleteTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54290
I20260928 06:31:06.888638 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet c2fbf68dc40144c6bb4643f3214b1939 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57406
I20260928 06:31:06.888850 26968 tablet_replica.cc:330] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: stopping tablet replica
I20260928 06:31:06.888923 26968 raft_consensus.cc:2229] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:06.888972 26968 raft_consensus.cc:2258] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.889067 26967 tablet_replica.cc:330] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: stopping tablet replica
I20260928 06:31:06.889195 26967 raft_consensus.cc:2229] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:06.889213 26968 ts_tablet_manager.cc:1905] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:06.889338 26967 raft_consensus.cc:2258] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.889626 26967 ts_tablet_manager.cc:1905] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:06.891139 26967 ts_tablet_manager.cc:1918] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:31:06.891223 26967 log.cc:1196] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/c2fbf68dc40144c6bb4643f3214b1939
I20260928 06:31:06.891273 26968 ts_tablet_manager.cc:1918] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:31:06.891363 26968 log.cc:1196] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/c2fbf68dc40144c6bb4643f3214b1939
I20260928 06:31:06.891525 26967 ts_tablet_manager.cc:1939] T c2fbf68dc40144c6bb4643f3214b1939 P 27a90dc1a322482f92fbed6c21e88989: Deleting consensus metadata
I20260928 06:31:06.891654 26968 ts_tablet_manager.cc:1939] T c2fbf68dc40144c6bb4643f3214b1939 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting consensus metadata
I20260928 06:31:06.891849 26966 tablet_replica.cc:330] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: stopping tablet replica
I20260928 06:31:06.891908 26356 catalog_manager.cc:4841] TS 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629): tablet c2fbf68dc40144c6bb4643f3214b1939 (table test-table [id=e510e88c277940c392d81c32f6ded3f3]) successfully deleted
I20260928 06:31:06.891932 26966 raft_consensus.cc:2229] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:06.891980 26966 raft_consensus.cc:2258] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.892056 26355 catalog_manager.cc:4841] TS 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535): tablet c2fbf68dc40144c6bb4643f3214b1939 (table test-table [id=e510e88c277940c392d81c32f6ded3f3]) successfully deleted
I20260928 06:31:06.893118 26966 ts_tablet_manager.cc:1905] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:06.894481 26966 ts_tablet_manager.cc:1918] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:31:06.894549 26966 log.cc:1196] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/c2fbf68dc40144c6bb4643f3214b1939
I20260928 06:31:06.894767 26966 ts_tablet_manager.cc:1939] T c2fbf68dc40144c6bb4643f3214b1939 P d731b2fc5d284af28b8a47a53844e393: Deleting consensus metadata
I20260928 06:31:06.895104 26356 catalog_manager.cc:4841] TS d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355): tablet c2fbf68dc40144c6bb4643f3214b1939 (table test-table [id=e510e88c277940c392d81c32f6ded3f3]) successfully deleted
Sep 28 06:31:06 dist-test-slave-38pg krb5kdc[26326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:31:06 dist-test-slave-38pg krb5kdc[26326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:06.917009 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.906544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.24.11.190:37025)
0928 06:31:06.906678 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:06.906680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:06.906697 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:31:06.906768 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:06.906769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:06.906780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:06.906851 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:06.907440 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.907951 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.908662 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.908847 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.908895 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.914538 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:06.914539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:06.914540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:06.914547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:06.915946 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.916372 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.916373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.916373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.916439 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:06.916605 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:06.916605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:06.916606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:06.916714 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:06.916789 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:06.916870 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:31:06.916936 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260928 06:31:06.918704 26371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:06.918874 26371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:06.920120 26648 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 (DEFAULT_TABLE table=test-table [id=696b0d94995a49609ea12a2b8fcbf37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.920106 26508 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 (DEFAULT_TABLE table=test-table [id=696b0d94995a49609ea12a2b8fcbf37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.920248 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6d0d2615344d48c53d810ac6f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.920271 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6d0d2615344d48c53d810ac6f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.921237 26977 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Bootstrap starting.
I20260928 06:31:06.921388 26978 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Bootstrap starting.
I20260928 06:31:06.921418 26788 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 (DEFAULT_TABLE table=test-table [id=696b0d94995a49609ea12a2b8fcbf37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.921569 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6d0d2615344d48c53d810ac6f87b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.921682 26977 tablet_bootstrap.cc:654] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.921764 26978 tablet_bootstrap.cc:654] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.922879 26981 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Bootstrap starting.
I20260928 06:31:06.922969 26978 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: No bootstrap required, opened a new log
I20260928 06:31:06.922981 26977 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: No bootstrap required, opened a new log
I20260928 06:31:06.923022 26977 ts_tablet_manager.cc:1397] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:06.923043 26978 ts_tablet_manager.cc:1397] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:31:06.923148 26977 raft_consensus.cc:348] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.923215 26977 raft_consensus.cc:374] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.923159 26978 raft_consensus.cc:348] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.923241 26977 raft_consensus.cc:729] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.923254 26978 raft_consensus.cc:374] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.923293 26978 raft_consensus.cc:729] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.923292 26977 consensus_queue.cc:260] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.923331 26981 tablet_bootstrap.cc:654] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.923365 26978 consensus_queue.cc:260] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.923393 26977 ts_tablet_manager.cc:1428] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.923550 26978 ts_tablet_manager.cc:1428] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.924217 26981 tablet_bootstrap.cc:492] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: No bootstrap required, opened a new log
I20260928 06:31:06.924299 26981 ts_tablet_manager.cc:1397] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.924438 26981 raft_consensus.cc:348] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.924512 26981 raft_consensus.cc:374] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.924536 26981 raft_consensus.cc:729] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.924592 26981 consensus_queue.cc:260] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.924683 26981 ts_tablet_manager.cc:1428] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.933722 26885 raft_consensus.cc:482] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:06.933837 26885 raft_consensus.cc:504] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.934024 26885 leader_election.cc:288] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:06.934175 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6d0d2615344d48c53d810ac6f87b2" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393" is_pre_election: true
I20260928 06:31:06.934187 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6d0d2615344d48c53d810ac6f87b2" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" is_pre_election: true
I20260928 06:31:06.934304 26802 raft_consensus.cc:2454] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 0.
I20260928 06:31:06.934314 26528 raft_consensus.cc:2454] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 0.
I20260928 06:31:06.934465 26600 leader_election.cc:302] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a90dc1a322482f92fbed6c21e88989, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:06.934561 26885 raft_consensus.cc:2781] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:06.934615 26885 raft_consensus.cc:482] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:06.934639 26885 raft_consensus.cc:3037] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:06.935310 26885 raft_consensus.cc:504] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.935453 26885 leader_election.cc:288] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 election: Requested vote from peers 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:06.935613 26528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6d0d2615344d48c53d810ac6f87b2" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17140b4cafc847f1a0780e5b4a2f38ba"
I20260928 06:31:06.935695 26528 raft_consensus.cc:3037] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:06.935669 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c6d0d2615344d48c53d810ac6f87b2" candidate_uuid: "27a90dc1a322482f92fbed6c21e88989" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393"
I20260928 06:31:06.935765 26802 raft_consensus.cc:3037] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:06.936364 26802 raft_consensus.cc:2454] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 1.
I20260928 06:31:06.936429 26528 raft_consensus.cc:2454] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a90dc1a322482f92fbed6c21e88989 in term 1.
I20260928 06:31:06.936575 26600 leader_election.cc:302] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a90dc1a322482f92fbed6c21e88989, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:06.936672 26885 raft_consensus.cc:2781] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:06.936754 26885 raft_consensus.cc:686] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 1 LEADER]: Becoming Leader. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Running, Role: LEADER
I20260928 06:31:06.936812 26885 consensus_queue.cc:237] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.937434 26371 catalog_manager.cc:5495] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "27a90dc1a322482f92fbed6c21e88989" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:06.944065 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:06.941793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.24.11.130:34629)
0928 06:31:06.941952 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:06.941954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:06.941962 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:06.941983 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:06.941983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:06.941998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:06.942068 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:06.942178 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.942653 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.943465 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:06.943663 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:06.943728 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:06.943835 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:06.943861 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:06.943897 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:06.943934 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:31:06.943952 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":114,"threads_started":1}
I20260928 06:31:06.946445 26528 raft_consensus.cc:1261] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Refusing update from remote peer 27a90dc1a322482f92fbed6c21e88989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:06.946445 26802 raft_consensus.cc:1261] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Refusing update from remote peer 27a90dc1a322482f92fbed6c21e88989: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:06.946643 26958 consensus_queue.cc:1035] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:06.946733 26885 consensus_queue.cc:1035] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:06.958659 26370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:06.958770 26370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:06.959395 26370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Sending DeleteTablet for 3 replicas of tablet 17c6d0d2615344d48c53d810ac6f87b2
I20260928 06:31:06.959668 26648 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:47452
I20260928 06:31:06.959693 26788 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54290
I20260928 06:31:06.959803 26966 tablet_replica.cc:330] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: stopping tablet replica
I20260928 06:31:06.959790 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6d0d2615344d48c53d810ac6f87b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:06 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57406
I20260928 06:31:06.959864 26966 raft_consensus.cc:2229] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:06.959877 26967 tablet_replica.cc:330] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: stopping tablet replica
I20260928 06:31:06.959875 26968 tablet_replica.cc:330] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: stopping tablet replica
I20260928 06:31:06.959910 26966 raft_consensus.cc:2258] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.959923 26967 raft_consensus.cc:2229] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:06.959930 26968 raft_consensus.cc:2229] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:06.959969 26968 raft_consensus.cc:2258] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.959976 26967 raft_consensus.cc:2258] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:06.960309 26967 ts_tablet_manager.cc:1905] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:06.960260 26370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:31:06.960332 26968 ts_tablet_manager.cc:1905] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:06.960412 26370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:06.960896 26966 ts_tablet_manager.cc:1905] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:06.961681 26508 tablet_service.cc:1467] Processing CreateTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 (DEFAULT_TABLE table=test-table [id=34168cff138542fc8062ed4ff4a61990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.961795 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace656c16d9e41dcaf2ed59b99bea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.962131 26968 ts_tablet_manager.cc:1918] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:06.962193 26968 log.cc:1196] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/17c6d0d2615344d48c53d810ac6f87b2
I20260928 06:31:06.962175 26967 ts_tablet_manager.cc:1918] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:06.962230 26967 log.cc:1196] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/17c6d0d2615344d48c53d810ac6f87b2
I20260928 06:31:06.962425 26968 ts_tablet_manager.cc:1939] T 17c6d0d2615344d48c53d810ac6f87b2 P 17140b4cafc847f1a0780e5b4a2f38ba: Deleting consensus metadata
I20260928 06:31:06.962473 26967 ts_tablet_manager.cc:1939] T 17c6d0d2615344d48c53d810ac6f87b2 P 27a90dc1a322482f92fbed6c21e88989: Deleting consensus metadata
I20260928 06:31:06.962507 26966 ts_tablet_manager.cc:1918] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:06.962555 26966 log.cc:1196] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/17c6d0d2615344d48c53d810ac6f87b2
I20260928 06:31:06.962754 26966 ts_tablet_manager.cc:1939] T 17c6d0d2615344d48c53d810ac6f87b2 P d731b2fc5d284af28b8a47a53844e393: Deleting consensus metadata
I20260928 06:31:06.963055 26355 catalog_manager.cc:4841] TS 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535): tablet 17c6d0d2615344d48c53d810ac6f87b2 (table test-table [id=696b0d94995a49609ea12a2b8fcbf37e]) successfully deleted
I20260928 06:31:06.963205 26978 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba: Bootstrap starting.
I20260928 06:31:06.963228 26356 catalog_manager.cc:4841] TS 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629): tablet 17c6d0d2615344d48c53d810ac6f87b2 (table test-table [id=696b0d94995a49609ea12a2b8fcbf37e]) successfully deleted
I20260928 06:31:06.963308 26356 catalog_manager.cc:4841] TS d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355): tablet 17c6d0d2615344d48c53d810ac6f87b2 (table test-table [id=696b0d94995a49609ea12a2b8fcbf37e]) successfully deleted
I20260928 06:31:06.963320 26788 tablet_service.cc:1467] Processing CreateTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 (DEFAULT_TABLE table=test-table [id=34168cff138542fc8062ed4ff4a61990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.963486 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace656c16d9e41dcaf2ed59b99bea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.963474 26648 tablet_service.cc:1467] Processing CreateTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 (DEFAULT_TABLE table=test-table [id=34168cff138542fc8062ed4ff4a61990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:06.963554 26978 tablet_bootstrap.cc:654] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.963580 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace656c16d9e41dcaf2ed59b99bea539. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:06.964128 26978 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba: No bootstrap required, opened a new log
I20260928 06:31:06.964170 26978 ts_tablet_manager.cc:1397] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:06.964306 26978 raft_consensus.cc:348] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.964373 26978 raft_consensus.cc:374] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.964392 26978 raft_consensus.cc:729] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.964434 26978 consensus_queue.cc:260] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.964526 26978 ts_tablet_manager.cc:1428] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.964697 26977 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Bootstrap starting.
I20260928 06:31:06.964747 26981 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Bootstrap starting.
I20260928 06:31:06.965015 26977 tablet_bootstrap.cc:654] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.965050 26981 tablet_bootstrap.cc:654] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:06.965734 26977 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: No bootstrap required, opened a new log
I20260928 06:31:06.965794 26977 ts_tablet_manager.cc:1397] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.965925 26981 tablet_bootstrap.cc:492] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: No bootstrap required, opened a new log
I20260928 06:31:06.965970 26981 ts_tablet_manager.cc:1397] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:06.965943 26977 raft_consensus.cc:348] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.966068 26977 raft_consensus.cc:374] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.966094 26977 raft_consensus.cc:729] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a90dc1a322482f92fbed6c21e88989, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.966109 26981 raft_consensus.cc:348] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.966141 26977 consensus_queue.cc:260] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.966176 26981 raft_consensus.cc:374] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:06.966254 26977 ts_tablet_manager.cc:1428] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:06.966202 26981 raft_consensus.cc:729] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d731b2fc5d284af28b8a47a53844e393, State: Initialized, Role: FOLLOWER
I20260928 06:31:06.967250 26981 consensus_queue.cc:260] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:06.967381 26981 ts_tablet_manager.cc:1428] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:07.393296 26982 raft_consensus.cc:482] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:07.393407 26982 raft_consensus.cc:504] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:07.393606 26982 leader_election.cc:288] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:07.393836 26667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace656c16d9e41dcaf2ed59b99bea539" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989" is_pre_election: true
I20260928 06:31:07.393857 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace656c16d9e41dcaf2ed59b99bea539" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393" is_pre_election: true
I20260928 06:31:07.393950 26802 raft_consensus.cc:2454] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 0.
I20260928 06:31:07.393955 26667 raft_consensus.cc:2454] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 0.
I20260928 06:31:07.394120 26460 leader_election.cc:302] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, 27a90dc1a322482f92fbed6c21e88989; no voters:
I20260928 06:31:07.394222 26982 raft_consensus.cc:2781] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:07.394289 26982 raft_consensus.cc:482] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:07.394322 26982 raft_consensus.cc:3037] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:07.395038 26982 raft_consensus.cc:504] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:07.395182 26982 leader_election.cc:288] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 election: Requested vote from peers 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629), d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355)
I20260928 06:31:07.395392 26802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace656c16d9e41dcaf2ed59b99bea539" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d731b2fc5d284af28b8a47a53844e393"
I20260928 06:31:07.395392 26667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ace656c16d9e41dcaf2ed59b99bea539" candidate_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a90dc1a322482f92fbed6c21e88989"
I20260928 06:31:07.395483 26802 raft_consensus.cc:3037] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:07.395483 26667 raft_consensus.cc:3037] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:07.396270 26667 raft_consensus.cc:2454] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 1.
I20260928 06:31:07.396270 26802 raft_consensus.cc:2454] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17140b4cafc847f1a0780e5b4a2f38ba in term 1.
I20260928 06:31:07.396443 26460 leader_election.cc:302] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17140b4cafc847f1a0780e5b4a2f38ba, d731b2fc5d284af28b8a47a53844e393; no voters:
I20260928 06:31:07.396538 26982 raft_consensus.cc:2781] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:07.396653 26982 raft_consensus.cc:686] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [term 1 LEADER]: Becoming Leader. State: Replica: 17140b4cafc847f1a0780e5b4a2f38ba, State: Running, Role: LEADER
I20260928 06:31:07.396698 26982 consensus_queue.cc:237] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } }
I20260928 06:31:07.397322 26370 catalog_manager.cc:5495] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129). New cstate: current_term: 1 leader_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17140b4cafc847f1a0780e5b4a2f38ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:07.412000 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:07.409911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57420 (local address 127.24.11.129:33535)
0928 06:31:07.410067 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:07.410069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:07.410076 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:07.410094 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:07.410094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:07.410103 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:07.410158 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:07.410312 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.410775 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.411421 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:07.411610 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:07.411658 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:07.411753 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:07.411776 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:07.411808 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:07.411840 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:31:07.411855 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":107,"threads_started":1}
I20260928 06:31:07.414158 26802 raft_consensus.cc:1261] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Refusing update from remote peer 17140b4cafc847f1a0780e5b4a2f38ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:07.414168 26667 raft_consensus.cc:1261] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Refusing update from remote peer 17140b4cafc847f1a0780e5b4a2f38ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:07.414362 26982 consensus_queue.cc:1035] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "27a90dc1a322482f92fbed6c21e88989" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:07.414490 26994 consensus_queue.cc:1035] T ace656c16d9e41dcaf2ed59b99bea539 P 17140b4cafc847f1a0780e5b4a2f38ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d731b2fc5d284af28b8a47a53844e393" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:31:07.418617 26950 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:31:08.935510 26371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:08.935629 26371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:08.936511 26371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf86cf5162e476194f820d59582f78d: Sending DeleteTablet for 3 replicas of tablet ace656c16d9e41dcaf2ed59b99bea539
I20260928 06:31:08.936745 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:08 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57406
I20260928 06:31:08.936784 26648 tablet_service.cc:1514] Processing DeleteTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:08 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:47452
I20260928 06:31:08.936810 26788 tablet_service.cc:1514] Processing DeleteTablet for tablet ace656c16d9e41dcaf2ed59b99bea539 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:08 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54290
I20260928 06:31:08.936894 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 26443
I20260928 06:31:08.937768 27024 tablet_replica.cc:330] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: stopping tablet replica
I20260928 06:31:08.937850 27024 raft_consensus.cc:2229] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:08.937888 27024 raft_consensus.cc:2258] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:08.937989 27023 tablet_replica.cc:330] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: stopping tablet replica
I20260928 06:31:08.938051 27023 raft_consensus.cc:2229] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:08.938095 27023 raft_consensus.cc:2258] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:08.938370 27024 ts_tablet_manager.cc:1905] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:08.939055 27023 ts_tablet_manager.cc:1905] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:08.940361 27024 ts_tablet_manager.cc:1918] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:31:08.940429 27024 log.cc:1196] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/ace656c16d9e41dcaf2ed59b99bea539
I20260928 06:31:08.940641 27023 ts_tablet_manager.cc:1918] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:31:08.940650 27024 ts_tablet_manager.cc:1939] T ace656c16d9e41dcaf2ed59b99bea539 P 27a90dc1a322482f92fbed6c21e88989: Deleting consensus metadata
I20260928 06:31:08.940716 27023 log.cc:1196] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/ace656c16d9e41dcaf2ed59b99bea539
I20260928 06:31:08.941025 27023 ts_tablet_manager.cc:1939] T ace656c16d9e41dcaf2ed59b99bea539 P d731b2fc5d284af28b8a47a53844e393: Deleting consensus metadata
I20260928 06:31:08.941035 26356 catalog_manager.cc:4841] TS 27a90dc1a322482f92fbed6c21e88989 (127.24.11.130:34629): tablet ace656c16d9e41dcaf2ed59b99bea539 (table test-table [id=34168cff138542fc8062ed4ff4a61990]) successfully deleted
I20260928 06:31:08.941366 26356 catalog_manager.cc:4841] TS d731b2fc5d284af28b8a47a53844e393 (127.24.11.131:35355): tablet ace656c16d9e41dcaf2ed59b99bea539 (table test-table [id=34168cff138542fc8062ed4ff4a61990]) successfully deleted
W20260928 06:31:08.944271 26355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:33535 (error 108)
I20260928 06:31:08.944537 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 26583
I20260928 06:31:08.945272 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:08.944628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33535 (local address 127.0.0.1:57426)
0928 06:31:08.945175 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:08.945226 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:33535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":58,"threads_started":1}
W20260928 06:31:08.945359 26355 catalog_manager.cc:4584] TS 17140b4cafc847f1a0780e5b4a2f38ba (127.24.11.129:33535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ace656c16d9e41dcaf2ed59b99bea539: Network error: Client connection negotiation failed: client connection to 127.24.11.129:33535: connect: Connection refused (error 111)
I20260928 06:31:08.950488 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 26723
I20260928 06:31:08.957243 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 26339
2026-09-28T06:31:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6299 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:09 dist-test-slave-38pg krb5kdc[27029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:09 dist-test-slave-38pg krb5kdc[27029](info): set up 2 sockets
Sep 28 06:31:09 dist-test-slave-38pg krb5kdc[27029](info): commencing operation
krb5kdc: starting...
W20260928 06:31:09.994803 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:10.111927 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35351
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:10.191555 27042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:10.191761 27042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:10.191784 27042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:10.192962 27042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:10.193006 27042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:10.193020 27042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:10.193033 27042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:10.193045 27042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:10.194595 27042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:10.194867 27042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:10.195117 27042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:10.200307 27042 server_base.cc:1034] running on GCE node
W20260928 06:31:10.200248 27047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.200314 27048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.200265 27050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:10.200804 27042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:10.201088 27042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:10.202251 27042 hybrid_clock.cc:648] HybridClock initialized: now 1790577070202228 us; error 46 us; skew 500 ppm
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:10.204672 27042 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:10.205111 27042 webserver.cc:460] Webserver started at http://127.24.11.190:44155/ using document root <none> and password file <none>
I20260928 06:31:10.205355 27042 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:10.205425 27042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:10.205595 27042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:10.206708 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "4d5363a90d4041a28a2135eaf5580cf0"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "2856befdd48e17439475b954cb465d3d"
server_key_iv: "50cec69f5f4b68b66b0e3e8bd9fbcad7"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.207086 27042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "4d5363a90d4041a28a2135eaf5580cf0"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "2856befdd48e17439475b954cb465d3d"
server_key_iv: "50cec69f5f4b68b66b0e3e8bd9fbcad7"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.208457 27042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:10.209190 27057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.209368 27042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:10.209466 27042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "4d5363a90d4041a28a2135eaf5580cf0"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "2856befdd48e17439475b954cb465d3d"
server_key_iv: "50cec69f5f4b68b66b0e3e8bd9fbcad7"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.209571 27042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:10.231418 27042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:10.233320 27042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:10.233494 27042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:10.237367 27042 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:35351
I20260928 06:31:10.237440 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:35351 every 8 connection(s)
I20260928 06:31:10.237826 27042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:10.238672 27110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:10.240922 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0: Bootstrap starting.
I20260928 06:31:10.241518 27110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:10.241832 27110 log.cc:824] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:10.242616 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0: No bootstrap required, opened a new log
I20260928 06:31:10.244201 27110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } }
I20260928 06:31:10.244342 27110 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:10.244366 27110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5363a90d4041a28a2135eaf5580cf0, State: Initialized, Role: FOLLOWER
I20260928 06:31:10.244545 27110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } }
I20260928 06:31:10.244611 27110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:10.244632 27110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:10.244678 27110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:10.245189 27110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } }
I20260928 06:31:10.245297 27110 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5363a90d4041a28a2135eaf5580cf0; no voters:
I20260928 06:31:10.245471 27110 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:10.245532 27113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:10.245714 27113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d5363a90d4041a28a2135eaf5580cf0, State: Running, Role: LEADER
I20260928 06:31:10.245754 27110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:10.245800 27113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } }
I20260928 06:31:10.246181 27115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5363a90d4041a28a2135eaf5580cf0. Latest consensus state: current_term: 1 leader_uuid: "4d5363a90d4041a28a2135eaf5580cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } } }
I20260928 06:31:10.246261 27115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:10.246333 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5363a90d4041a28a2135eaf5580cf0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5363a90d4041a28a2135eaf5580cf0" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35351 } } }
I20260928 06:31:10.246415 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:10.246763 27121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:10.246901 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27042
I20260928 06:31:10.246995 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:10.247197 24622 external_mini_cluster.cc:1428] Setting key 027c94d7fea43d69be5f937ee16c7717
I20260928 06:31:10.247296 27121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:10.248936 27121 catalog_manager.cc:1347] Generated new cluster ID: f0496d33571248abb467a142ee64997b
I20260928 06:31:10.248989 27121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:10.259408 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.247654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.24.11.190:35351)
0928 06:31:10.248332 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:10.248342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:10.248376 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:31:10.248510 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:10.248516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:10.248560 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:10.248723 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:10.249558 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.250937 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.252028 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.252611 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.252705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.255653 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:10.255657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:10.255658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:10.255671 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:10.257743 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.258119 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.258122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.258123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.258245 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.258442 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.258442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.258443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.258773 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:10.258949 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:10.259020 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:31:10.259164 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":69,"threads_started":1}
I20260928 06:31:10.271257 27121 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:10.272112 27121 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:10.274904 27121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5363a90d4041a28a2135eaf5580cf0: Generated new TSK 0
I20260928 06:31:10.275092 27121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:10.304684 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:10.406373 27136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:10.406611 27136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:10.406652 27136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:10.407893 27136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:10.407961 27136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:10.408006 27136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:10.409468 27136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:10.409754 27136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:10.409999 27136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:10.415220 27136 server_base.cc:1034] running on GCE node
W20260928 06:31:10.415184 27144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.415197 27142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.415179 27141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:10.415589 27136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:10.415831 27136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:10.417001 27136 hybrid_clock.cc:648] HybridClock initialized: now 1790577070416998 us; error 38 us; skew 500 ppm
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:10.419631 27136 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:10.420125 27136 webserver.cc:460] Webserver started at http://127.24.11.129:46219/ using document root <none> and password file <none>
I20260928 06:31:10.420364 27136 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:10.420428 27136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:10.420563 27136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:10.421691 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "df126a96ce4c4509b52c165dbf0c5e11"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "0eb31f1d2d1d46c185a1868f83e98aa1"
server_key_iv: "c666a0c884cf7940d620827265e48689"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.422055 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "df126a96ce4c4509b52c165dbf0c5e11"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "0eb31f1d2d1d46c185a1868f83e98aa1"
server_key_iv: "c666a0c884cf7940d620827265e48689"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.423373 27136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:10.424119 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.424309 27136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:10.424408 27136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "df126a96ce4c4509b52c165dbf0c5e11"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "0eb31f1d2d1d46c185a1868f83e98aa1"
server_key_iv: "c666a0c884cf7940d620827265e48689"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.424480 27136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:10.432870 27136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:10.434733 27136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:10.434906 27136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:10.435142 27136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:10.435451 27136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:10.435503 27136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.435542 27136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:10.435571 27136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.441514 27136 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:36483
I20260928 06:31:10.441579 27264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:36483 every 8 connection(s)
I20260928 06:31:10.441901 27136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:10.450546 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27136
I20260928 06:31:10.450659 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:10.450816 24622 external_mini_cluster.cc:1428] Setting key 2499353707376cebaf8baca5a9c3a08b
I20260928 06:31:10.451833 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.442662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:40821 (local address 127.24.11.190:35351)
0928 06:31:10.442732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:10.442745 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:10.442937 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:31:10.443484 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:10.443486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:10.443503 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:10.443559 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:10.444521 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.445029 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.446368 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.446588 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.446661 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.448796 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:10.448797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:10.448798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:10.448805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:10.450187 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.450649 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.450650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.450651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.450718 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.451136 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.451138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.451138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.451266 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:10.451346 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:10.451619 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:31:10.451715 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:31:10.452000 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.442640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35351 (local address 127.24.11.129:40821)
0928 06:31:10.442809 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:10.442851 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:10.443360 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:10.443629 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:10.443633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:10.444208 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:10.444409 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.444422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.445127 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.445128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.446143 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.446146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.446670 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.446671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.446775 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.447310 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:10.447332 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:10.448698 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:10.450276 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.450282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.450283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.450533 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.450836 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.450837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.450837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.450972 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.451338 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:10.451345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:10.451480 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:31:10.451595 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:31:10.451760 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260928 06:31:10.452549 27265 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35351
I20260928 06:31:10.452666 27265 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:10.452878 27265 heartbeater.cc:511] Master 127.24.11.190:35351 requested a full tablet report, sending...
I20260928 06:31:10.453491 27074 ts_manager.cc:194] Registered new tserver with Master: df126a96ce4c4509b52c165dbf0c5e11 (127.24.11.129:36483)
I20260928 06:31:10.454263 27074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:40821
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:10.475756 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:10.556798 27270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:10.557019 27270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:10.557045 27270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:10.558362 27270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:10.558421 27270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:10.558450 27270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:10.559906 27270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:10.560158 27270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:10.560400 27270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:10.565429 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.565424 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.565424 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:10.565655 27270 server_base.cc:1034] running on GCE node
I20260928 06:31:10.565879 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:10.566126 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:10.567293 27270 hybrid_clock.cc:648] HybridClock initialized: now 1790577070567276 us; error 36 us; skew 500 ppm
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:10.569923 27270 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:10.570384 27270 webserver.cc:460] Webserver started at http://127.24.11.130:36057/ using document root <none> and password file <none>
I20260928 06:31:10.570577 27270 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:10.570618 27270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:10.570706 27270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:10.571786 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "55ea307c3450446383c7d29265b3c6a4"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "dd307dcf0b86bb9d24c247d22f2a06c1"
server_key_iv: "01bc6e02e79f1601a1382d0b98ba719d"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.572093 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "55ea307c3450446383c7d29265b3c6a4"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "dd307dcf0b86bb9d24c247d22f2a06c1"
server_key_iv: "01bc6e02e79f1601a1382d0b98ba719d"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.573365 27270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:10.574159 27285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.574314 27270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.574414 27270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "55ea307c3450446383c7d29265b3c6a4"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "dd307dcf0b86bb9d24c247d22f2a06c1"
server_key_iv: "01bc6e02e79f1601a1382d0b98ba719d"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.574486 27270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:10.607535 27270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:10.610020 27270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:10.610191 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:10.610440 27270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:10.610800 27270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:10.610849 27270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.610893 27270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:10.610920 27270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.616704 27270 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:44343
I20260928 06:31:10.616815 27398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:44343 every 8 connection(s)
I20260928 06:31:10.617116 27270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:10.620957 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27270
I20260928 06:31:10.621085 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:10.621268 24622 external_mini_cluster.cc:1428] Setting key f71a57e521ac91b70ee86df805002ceb
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:10.626716 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.617827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35501 (local address 127.24.11.190:35351)
0928 06:31:10.617877 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:10.617879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:10.618075 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:31:10.618561 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:10.618562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:10.618573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:10.618630 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:10.619513 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.620011 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.621051 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.621312 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.621389 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.623782 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:10.623783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:10.623784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:10.623793 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:10.625239 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.625745 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.625747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.625747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.625806 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.626062 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.626064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.626064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.626181 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:10.626256 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:10.626538 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 06:31:10.626623 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:10.626958 27401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.617801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35351 (local address 127.24.11.130:35501)
0928 06:31:10.617955 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:10.617993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:10.618464 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:10.618683 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:10.618686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:10.619235 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:10.619438 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.619443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.620093 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.620094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.620948 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.620950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.621434 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.621436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.621602 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.622141 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:10.622150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:10.623690 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:10.625343 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.625351 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.625354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.625657 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.625871 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.625872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.625872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.625995 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.626248 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:10.626254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:10.626406 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:31:10.626519 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:31:10.626682 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 06:31:10.627476 27399 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35351
I20260928 06:31:10.627584 27399 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:10.627756 27399 heartbeater.cc:511] Master 127.24.11.190:35351 requested a full tablet report, sending...
I20260928 06:31:10.628175 27074 ts_manager.cc:194] Registered new tserver with Master: 55ea307c3450446383c7d29265b3c6a4 (127.24.11.130:44343)
I20260928 06:31:10.628710 27074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:35501
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:10.648444 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:10.731307 27404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:10.731513 27404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:10.731536 27404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:10.732654 27404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:10.732707 27404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:10.732735 27404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:10.734170 27404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:10.734359 27404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:10.734601 27404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:10.739575 27412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.739579 27410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:10.739627 27409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:10.739595 27404 server_base.cc:1034] running on GCE node
I20260928 06:31:10.740043 27404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:10.740247 27404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:10.741451 27404 hybrid_clock.cc:648] HybridClock initialized: now 1790577070741443 us; error 40 us; skew 500 ppm
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:10.744225 27404 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:10.744717 27404 webserver.cc:460] Webserver started at http://127.24.11.131:34173/ using document root <none> and password file <none>
I20260928 06:31:10.744980 27404 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:10.745051 27404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:10.745190 27404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:10.746429 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "0c30ddddfa0c40268fe883a9a2349b46"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "b716f2e6bde6bb8e0619ba2d6e6164f1"
server_key_iv: "959053c0154534fa32864754319bae57"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.746862 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "0c30ddddfa0c40268fe883a9a2349b46"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "b716f2e6bde6bb8e0619ba2d6e6164f1"
server_key_iv: "959053c0154534fa32864754319bae57"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.748210 27404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:10.748994 27419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.749166 27404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:10.749264 27404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "0c30ddddfa0c40268fe883a9a2349b46"
format_stamp: "Formatted at 2026-09-28 06:31:10 on dist-test-slave-38pg"
server_key: "b716f2e6bde6bb8e0619ba2d6e6164f1"
server_key_iv: "959053c0154534fa32864754319bae57"
server_key_version: "encryptionkey@0"
I20260928 06:31:10.749353 27404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:10.771672 27404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:10.774101 27404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:10.774273 27404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:10.774539 27404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:10.774932 27404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:10.775053 27404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.775106 27404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:10.775128 27404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:10.781328 27404 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:33549
I20260928 06:31:10.781370 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:33549 every 8 connection(s)
I20260928 06:31:10.781766 27404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:10.783493 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27404
I20260928 06:31:10.783614 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:10.783783 24622 external_mini_cluster.cc:1428] Setting key 9d3cd8cc97cc91a42c339007444b4edb
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:10.791422 27132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.782477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52079 (local address 127.24.11.190:35351)
0928 06:31:10.782559 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:10.782561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:10.782852 (+ 291us) server_negotiation.cc:407] Connection header received
0928 06:31:10.783414 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:10.783415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:10.783428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:10.783482 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:10.784835 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.785327 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.786440 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.786633 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.786717 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.788981 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:10.788983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:10.788983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:10.788990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:10.790269 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.790646 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.790647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.790647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.790695 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:10.790898 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:10.790898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:10.790899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:10.790997 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:10.791046 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:10.791277 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:31:10.791354 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:10.791599 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:10.782464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35351 (local address 127.24.11.131:52079)
0928 06:31:10.782698 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:10.782739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:10.783264 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:10.783570 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:10.783575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:10.784390 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:10.784665 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.784671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.785427 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.785428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.786333 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:10.786335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:10.786730 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:10.786731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:10.786882 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:10.787416 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:10.787425 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:10.788866 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:10.790341 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.790347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.790349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.790582 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.790749 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:10.790749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:10.790750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:10.790848 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:10.791047 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:10.791050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:10.791172 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:31:10.791254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:31:10.791378 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
I20260928 06:31:10.792079 27533 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35351
I20260928 06:31:10.792174 27533 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:10.792320 27533 heartbeater.cc:511] Master 127.24.11.190:35351 requested a full tablet report, sending...
I20260928 06:31:10.792667 27074 ts_manager.cc:194] Registered new tserver with Master: 0c30ddddfa0c40268fe883a9a2349b46 (127.24.11.131:33549)
I20260928 06:31:10.793046 27074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:52079
I20260928 06:31:10.795521 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:31:10.806689 27132 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:10.805671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.24.11.190:35351)
0928 06:31:10.805720 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:10.805722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:10.805879 (+ 157us) server_negotiation.cc:407] Connection header received
0928 06:31:10.805908 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:10.805909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:10.805919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:10.806143 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:10.806629 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58096: BlockingRecv error: recv got EOF from 127.0.0.1:58096 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:10.807602 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27136
I20260928 06:31:10.812042 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27270
I20260928 06:31:10.816352 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27404
I20260928 06:31:10.820674 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27042
2026-09-28T06:31:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1862 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27545](info): set up 2 sockets
Sep 28 06:31:10 dist-test-slave-38pg krb5kdc[27545](info): commencing operation
krb5kdc: starting...
W20260928 06:31:11.855252 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:11 dist-test-slave-38pg krb5kdc[27545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:11.971907 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37447
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:12.052676 27558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:12.052901 27558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:12.052923 27558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:12.054116 27558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:12.054158 27558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:12.054173 27558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:12.054185 27558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:12.054198 27558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:12.055711 27558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:12.055967 27558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:12.056217 27558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:12.060981 27563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:12.061025 27566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:12.061087 27564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:12.061494 27558 server_base.cc:1034] running on GCE node
I20260928 06:31:12.061785 27558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:12.062069 27558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:12.063222 27558 hybrid_clock.cc:648] HybridClock initialized: now 1790577072063215 us; error 33 us; skew 500 ppm
Sep 28 06:31:12 dist-test-slave-38pg krb5kdc[27545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577072, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:12.076239 27558 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:12.076761 27558 webserver.cc:460] Webserver started at http://127.24.11.190:34265/ using document root <none> and password file <none>
I20260928 06:31:12.076988 27558 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:12.077052 27558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:12.077205 27558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:12.078393 27558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "7e05e6679ddd472fb7538af1d1b2646b"
format_stamp: "Formatted at 2026-09-28 06:31:12 on dist-test-slave-38pg"
server_key: "34f2add0e41577d24ad5c2700cb72b99"
server_key_iv: "b4a33e0bf60d48da61def104508bd4fc"
server_key_version: "encryptionkey@0"
I20260928 06:31:12.078763 27558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "7e05e6679ddd472fb7538af1d1b2646b"
format_stamp: "Formatted at 2026-09-28 06:31:12 on dist-test-slave-38pg"
server_key: "34f2add0e41577d24ad5c2700cb72b99"
server_key_iv: "b4a33e0bf60d48da61def104508bd4fc"
server_key_version: "encryptionkey@0"
I20260928 06:31:12.080154 27558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:12.080953 27573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:12.081158 27558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:12.081257 27558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "7e05e6679ddd472fb7538af1d1b2646b"
format_stamp: "Formatted at 2026-09-28 06:31:12 on dist-test-slave-38pg"
server_key: "34f2add0e41577d24ad5c2700cb72b99"
server_key_iv: "b4a33e0bf60d48da61def104508bd4fc"
server_key_version: "encryptionkey@0"
I20260928 06:31:12.081336 27558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:12.104792 27558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:12.107208 27558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:12.107370 27558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:12.111368 27558 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:37447
I20260928 06:31:12.111404 27625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:37447 every 8 connection(s)
I20260928 06:31:12.111881 27558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:12.112515 27626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:12.114826 27626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b: Bootstrap starting.
I20260928 06:31:12.115356 27626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:12.115588 27626 log.cc:824] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:12.116180 27626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b: No bootstrap required, opened a new log
I20260928 06:31:12.116856 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27558
I20260928 06:31:12.116967 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:12.117137 24622 external_mini_cluster.cc:1428] Setting key 1ed887face3f5df860ffe85a269d01b3
I20260928 06:31:12.118695 27626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } }
I20260928 06:31:12.118878 27626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:12.118919 27626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e05e6679ddd472fb7538af1d1b2646b, State: Initialized, Role: FOLLOWER
I20260928 06:31:12.119073 27626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } }
I20260928 06:31:12.119138 27626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:12.119158 27626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:12.119221 27626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:12.119791 27626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } }
I20260928 06:31:12.119894 27626 leader_election.cc:302] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e05e6679ddd472fb7538af1d1b2646b; no voters:
I20260928 06:31:12.120038 27626 leader_election.cc:288] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:12.120095 27631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:12.120244 27631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [term 1 LEADER]: Becoming Leader. State: Replica: 7e05e6679ddd472fb7538af1d1b2646b, State: Running, Role: LEADER
I20260928 06:31:12.120276 27626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:12.120395 27631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } }
I20260928 06:31:12.120808 27632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e05e6679ddd472fb7538af1d1b2646b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } } }
I20260928 06:31:12.120882 27632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:12.121368 27633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e05e6679ddd472fb7538af1d1b2646b. Latest consensus state: current_term: 1 leader_uuid: "7e05e6679ddd472fb7538af1d1b2646b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e05e6679ddd472fb7538af1d1b2646b" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37447 } } }
I20260928 06:31:12.121449 27633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:12.121604 27638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:12.122900 27638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:12.124334 27638 catalog_manager.cc:1347] Generated new cluster ID: 756ce3a1c3a9497e89f47cb7b301b354
I20260928 06:31:12.124383 27638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:12 dist-test-slave-38pg krb5kdc[27545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:12.129277 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:12.117628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.24.11.190:37447)
0928 06:31:12.117813 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:12.117823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:12.117862 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:31:12.118654 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:12.118663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:12.118728 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:12.118876 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:12.119653 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:12.121049 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:12.122206 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:12.122656 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:12.122740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:12.125489 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:12.125493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:12.125494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:12.125509 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:12.127559 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:12.127944 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:12.127947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:12.127947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:12.128083 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:12.128297 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:12.128298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:12.128298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:12.128679 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:12.128838 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:12.128919 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:31:12.129059 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 06:31:12.129724 27638 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:12.130323 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:12.130388 27638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:12.136432 27638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e05e6679ddd472fb7538af1d1b2646b: Generated new TSK 0
I20260928 06:31:12.136665 27638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:12.144997 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:12.141349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50048 (local address 127.24.11.190:37447)
0928 06:31:12.141424 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:12.141426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:12.141522 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:31:12.141601 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:12.141602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:12.141612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:12.141665 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:12.142083 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:12.142561 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:12.143625 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:12.143822 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:12.143871 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:12.144542 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:12.144544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:12.144544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:12.144740 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:12.144755 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:12.144761 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:12.144805 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:12.144886 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:31:12.144910 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:31:12.147339 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27558
2026-09-28T06:31:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1323 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:12 dist-test-slave-38pg krb5kdc[27659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:12 dist-test-slave-38pg krb5kdc[27659](info): set up 2 sockets
Sep 28 06:31:12 dist-test-slave-38pg krb5kdc[27659](info): commencing operation
krb5kdc: starting...
W20260928 06:31:13.170506 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:13.286278 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41461
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:41461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:13.366978 27672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:13.367259 27672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:13.367305 27672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:13.368531 27672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:13.368603 27672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:13.368626 27672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:13.368647 27672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:13.368676 27672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:13.370234 27672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:41461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:13.370525 27672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:13.370759 27672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:13.376058 27678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.376156 27680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:13.376160 27672 server_base.cc:1034] running on GCE node
W20260928 06:31:13.376060 27677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:13.376543 27672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:13.376857 27672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:13.378047 27672 hybrid_clock.cc:648] HybridClock initialized: now 1790577073378030 us; error 40 us; skew 500 ppm
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:13.380576 27672 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:13.381093 27672 webserver.cc:460] Webserver started at http://127.24.11.190:46327/ using document root <none> and password file <none>
I20260928 06:31:13.381336 27672 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:13.381404 27672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:13.381595 27672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:13.382668 27672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "3e281955ff9a41d48e9c99f5594d5295"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "cb2bfa436c4afcf2e702907524b8a499"
server_key_iv: "5e96e03594f58f6cb2371fedf6db857b"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.383049 27672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "3e281955ff9a41d48e9c99f5594d5295"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "cb2bfa436c4afcf2e702907524b8a499"
server_key_iv: "5e96e03594f58f6cb2371fedf6db857b"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.384464 27672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:13.385238 27687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.385406 27672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:13.385505 27672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "3e281955ff9a41d48e9c99f5594d5295"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "cb2bfa436c4afcf2e702907524b8a499"
server_key_iv: "5e96e03594f58f6cb2371fedf6db857b"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.385634 27672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:13.399622 27672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:13.401629 27672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:13.401789 27672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:13.405939 27672 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:41461
I20260928 06:31:13.406011 27739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:41461 every 8 connection(s)
I20260928 06:31:13.406360 27672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:13.407351 27740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:13.409667 27740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295: Bootstrap starting.
I20260928 06:31:13.410229 27740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:13.410461 27740 log.cc:824] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:13.411108 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27672
I20260928 06:31:13.411224 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:13.411268 27740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295: No bootstrap required, opened a new log
I20260928 06:31:13.411417 24622 external_mini_cluster.cc:1428] Setting key e101d0694660d6d8cd28ba5f0e928eb3
I20260928 06:31:13.413816 27740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } }
I20260928 06:31:13.414002 27740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:13.414038 27740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e281955ff9a41d48e9c99f5594d5295, State: Initialized, Role: FOLLOWER
I20260928 06:31:13.414193 27740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } }
I20260928 06:31:13.414256 27740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:13.414278 27740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:13.414338 27740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:13.414956 27740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } }
I20260928 06:31:13.415058 27740 leader_election.cc:302] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e281955ff9a41d48e9c99f5594d5295; no voters:
I20260928 06:31:13.415205 27740 leader_election.cc:288] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:13.415301 27745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:13.415481 27740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:13.415526 27745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [term 1 LEADER]: Becoming Leader. State: Replica: 3e281955ff9a41d48e9c99f5594d5295, State: Running, Role: LEADER
I20260928 06:31:13.415719 27745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } }
I20260928 06:31:13.416064 27746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e281955ff9a41d48e9c99f5594d5295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } } }
I20260928 06:31:13.416139 27746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:13.416297 27747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e281955ff9a41d48e9c99f5594d5295. Latest consensus state: current_term: 1 leader_uuid: "3e281955ff9a41d48e9c99f5594d5295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e281955ff9a41d48e9c99f5594d5295" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41461 } } }
I20260928 06:31:13.416365 27747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:13.416802 27754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:13.417402 27754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:13.419773 27754 catalog_manager.cc:1347] Generated new cluster ID: f5b0798c1b104cbfb886084198aad18d
I20260928 06:31:13.419836 27754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:13.424924 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.412004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.24.11.190:41461)
0928 06:31:13.412193 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:13.412204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:13.412236 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:31:13.413764 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:13.413773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:13.413838 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:13.413992 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:13.414787 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.416348 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.417518 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.418141 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.418252 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.420810 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:13.420814 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:13.420815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:13.420827 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:13.423035 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.423532 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.423536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.423536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.423647 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.423856 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.423857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.423857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.424175 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:13.424332 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:13.424520 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:31:13.424675 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260928 06:31:13.445256 27754 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:13.445911 27754 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:13.449170 27754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e281955ff9a41d48e9c99f5594d5295: Generated new TSK 0
I20260928 06:31:13.449378 27754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:13.476862 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:13.560684 27766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:13.560978 27766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:13.561009 27766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:13.562253 27766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:13.562300 27766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:13.562353 27766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:13.563902 27766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:13.564168 27766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:13.564407 27766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:13.569501 27774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.569578 27772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.569623 27771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:13.569805 27766 server_base.cc:1034] running on GCE node
I20260928 06:31:13.570011 27766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:13.570225 27766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:13.571367 27766 hybrid_clock.cc:648] HybridClock initialized: now 1790577073571358 us; error 30 us; skew 500 ppm
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:13.574115 27766 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:13.574555 27766 webserver.cc:460] Webserver started at http://127.24.11.129:46193/ using document root <none> and password file <none>
I20260928 06:31:13.574800 27766 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:13.574846 27766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:13.574973 27766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:13.576045 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "7deab25b78234585aa8666899a4cae01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "a285dd8ba27181e3ad41ae34fe1f5752"
server_key_iv: "d2c4fa1eb9579dc9490aab0bc2ff3231"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.576429 27766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "7deab25b78234585aa8666899a4cae01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "a285dd8ba27181e3ad41ae34fe1f5752"
server_key_iv: "d2c4fa1eb9579dc9490aab0bc2ff3231"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.577811 27766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:31:13.578569 27781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.578766 27766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:13.578872 27766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "7deab25b78234585aa8666899a4cae01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "a285dd8ba27181e3ad41ae34fe1f5752"
server_key_iv: "d2c4fa1eb9579dc9490aab0bc2ff3231"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.578927 27766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:13.597163 27766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:13.599630 27766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:13.599753 27766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:13.599958 27766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:13.600287 27766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:13.600325 27766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.600365 27766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:13.600392 27766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.606319 27766 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:45447
I20260928 06:31:13.606412 27894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:45447 every 8 connection(s)
I20260928 06:31:13.606727 27766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:13.612432 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27766
I20260928 06:31:13.612567 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:13.612797 24622 external_mini_cluster.cc:1428] Setting key 88aff7a1885babc9876b841ed4357d78
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:13.618521 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.607602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58023 (local address 127.24.11.190:41461)
0928 06:31:13.607651 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:13.607653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:13.607890 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:31:13.608454 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:13.608455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:13.608468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:13.608521 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:13.609563 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.610392 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.611556 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.611795 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.611870 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.614729 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:13.614731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:13.614731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:13.614747 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:13.616852 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.617372 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.617374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.617374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.617447 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.617744 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.617746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.617746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.617931 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:13.618009 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:13.618273 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:31:13.618384 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:13.618541 27897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.607569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41461 (local address 127.24.11.129:58023)
0928 06:31:13.607754 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:13.607798 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:13.608339 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:13.608601 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:13.608605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:13.609221 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:13.609427 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.609433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.610499 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.610500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.611455 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.611457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.611878 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.611879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.612033 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.612658 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:13.612671 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:13.614635 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:13.616958 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.616965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.616968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.617293 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.617525 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.617557 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.617558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.617678 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.618002 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:13.618007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:13.618134 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:31:13.618223 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:31:13.618356 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260928 06:31:13.619087 27895 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41461
I20260928 06:31:13.619206 27895 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:13.619431 27895 heartbeater.cc:511] Master 127.24.11.190:41461 requested a full tablet report, sending...
I20260928 06:31:13.619993 27704 ts_manager.cc:194] Registered new tserver with Master: 7deab25b78234585aa8666899a4cae01 (127.24.11.129:45447)
I20260928 06:31:13.620618 27704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:58023
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:13.640986 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:13.725045 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:13.725260 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:13.725283 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:13.726485 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:13.726531 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:13.726558 27900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:13.728022 27900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:13.728266 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:13.728514 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:13.733613 27900 server_base.cc:1034] running on GCE node
W20260928 06:31:13.733570 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.733594 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.733592 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:13.734097 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:13.734383 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:13.735543 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790577073735525 us; error 42 us; skew 500 ppm
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:13.738142 27900 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:13.738642 27900 webserver.cc:460] Webserver started at http://127.24.11.130:44059/ using document root <none> and password file <none>
I20260928 06:31:13.738902 27900 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:13.738952 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:13.739078 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:13.740137 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "32115f8a55294b79a01eb9770d8a6a01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "bebd4412d932c0c7a683ad4de6002861"
server_key_iv: "073fd6f99126c1e4c6859854e3562574"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.740513 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "32115f8a55294b79a01eb9770d8a6a01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "bebd4412d932c0c7a683ad4de6002861"
server_key_iv: "073fd6f99126c1e4c6859854e3562574"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.741958 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:13.742733 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.742934 27900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:13.743054 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "32115f8a55294b79a01eb9770d8a6a01"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "bebd4412d932c0c7a683ad4de6002861"
server_key_iv: "073fd6f99126c1e4c6859854e3562574"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.743125 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:13.760017 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:13.762917 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:13.763082 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:13.763346 27900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:13.763670 27900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:13.763705 27900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.763756 27900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:13.763784 27900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.769922 27900 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33431
I20260928 06:31:13.769971 28028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33431 every 8 connection(s)
I20260928 06:31:13.770319 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:13.776183 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 27900
I20260928 06:31:13.776317 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:13.776526 24622 external_mini_cluster.cc:1428] Setting key 94976e38f318eaed8ca98767cc2a024b
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:13.781206 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.771109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:34959 (local address 127.24.11.190:41461)
0928 06:31:13.771155 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:13.771157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:13.771355 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:31:13.771845 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:13.771846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:13.771855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:13.771909 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:13.772882 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.773381 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.774546 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.774729 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.774792 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.777648 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:13.777649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:13.777650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:13.777658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:13.779333 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.779896 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.779897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.779897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.779979 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.780327 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.780328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.780328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.780502 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:13.780579 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:13.780959 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 06:31:13.781059 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:13.781286 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.771071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41461 (local address 127.24.11.130:34959)
0928 06:31:13.771227 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:13.771270 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:13.771762 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:13.771965 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:13.771969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:13.772580 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:13.772778 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.772784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.773492 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.773493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.774452 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.774455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.774857 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.774858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.775011 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.775591 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:13.775600 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:13.777521 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:13.779421 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.779430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.779433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.779814 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.780066 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.780067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.780067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.780240 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.780620 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:13.780627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:13.780802 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:31:13.780910 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:31:13.781064 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 06:31:13.782116 28029 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41461
I20260928 06:31:13.782212 28029 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:13.782366 28029 heartbeater.cc:511] Master 127.24.11.190:41461 requested a full tablet report, sending...
I20260928 06:31:13.782799 27704 ts_manager.cc:194] Registered new tserver with Master: 32115f8a55294b79a01eb9770d8a6a01 (127.24.11.130:33431)
I20260928 06:31:13.783191 27704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:34959
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:13.803920 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:13.885450 28034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:13.885730 28034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:13.885754 28034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:13.886869 28034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:13.886904 28034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:13.886930 28034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:13.888408 28034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:13.888618 28034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:13.888854 28034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:13.893899 28039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.893868 28040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:13.893970 28042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:13.893947 28034 server_base.cc:1034] running on GCE node
I20260928 06:31:13.894217 28034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:13.894467 28034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:13.895630 28034 hybrid_clock.cc:648] HybridClock initialized: now 1790577073895624 us; error 36 us; skew 500 ppm
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:13.898300 28034 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:13.898800 28034 webserver.cc:460] Webserver started at http://127.24.11.131:36935/ using document root <none> and password file <none>
I20260928 06:31:13.899034 28034 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:13.899083 28034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:13.899214 28034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:13.900326 28034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "8bef473c256a435ea253086d88ef9296"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "9a53997187f79514320df42de4dbc4b3"
server_key_iv: "1e86275a82becdde0dd40a556cd78485"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.900687 28034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "8bef473c256a435ea253086d88ef9296"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "9a53997187f79514320df42de4dbc4b3"
server_key_iv: "1e86275a82becdde0dd40a556cd78485"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.902155 28034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:13.903160 28049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.903401 28034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:13.903514 28034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "8bef473c256a435ea253086d88ef9296"
format_stamp: "Formatted at 2026-09-28 06:31:13 on dist-test-slave-38pg"
server_key: "9a53997187f79514320df42de4dbc4b3"
server_key_iv: "1e86275a82becdde0dd40a556cd78485"
server_key_version: "encryptionkey@0"
I20260928 06:31:13.903591 28034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:13.922119 28034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:13.924034 28034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:13.924201 28034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:13.924446 28034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:13.924825 28034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:13.924886 28034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.924929 28034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:13.924955 28034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:13.931056 28034 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:34405
I20260928 06:31:13.931106 28162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:34405 every 8 connection(s)
I20260928 06:31:13.931448 28034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:13.939044 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28034
I20260928 06:31:13.939147 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:13.939319 24622 external_mini_cluster.cc:1428] Setting key b079b35badddbf3e1827de07cef1ee99
I20260928 06:31:13.940743 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.932196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:46259 (local address 127.24.11.190:41461)
0928 06:31:13.932258 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:13.932260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:13.932439 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:31:13.932978 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:13.932978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:13.932988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:13.933055 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:13.934036 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.934525 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.935523 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.935707 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.935760 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.937816 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:13.937818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:13.937818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:13.937824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:13.939275 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.939699 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.939701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.939701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.939773 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.940046 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.940047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.940047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.940208 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:13.940280 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:13.940568 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:31:13.940653 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:31:13.940950 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.932162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41461 (local address 127.24.11.131:46259)
0928 06:31:13.932313 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:13.932355 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:13.932837 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:13.933112 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:13.933116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:13.933754 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:13.933958 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.933964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.934602 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.934603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.935447 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:13.935449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.935776 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.935776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:13.935904 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.936365 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:13.936376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:13.937722 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:13.939346 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.939355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.939357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.939615 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.939833 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:13.939834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:13.939834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:13.939973 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:13.940282 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:13.940290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:13.940433 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 06:31:13.940537 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:31:13.940696 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260928 06:31:13.941337 28163 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41461
I20260928 06:31:13.941454 28163 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:13.941651 28163 heartbeater.cc:511] Master 127.24.11.190:41461 requested a full tablet report, sending...
I20260928 06:31:13.942066 27704 ts_manager.cc:194] Registered new tserver with Master: 8bef473c256a435ea253086d88ef9296 (127.24.11.131:34405)
I20260928 06:31:13.942462 27704 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:46259
I20260928 06:31:13.951159 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:31:13 dist-test-slave-38pg krb5kdc[27659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577073, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:13.973881 27744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:13.966682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.24.11.190:41461)
0928 06:31:13.966747 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:13.966749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:13.966940 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:31:13.966971 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:13.966972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:13.966983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:13.967039 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:13.967636 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.968151 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.968952 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:13.969130 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:13.969182 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:13.971437 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:13.971439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:13.971439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:13.971446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:13.972860 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.973155 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.973156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.973156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.973208 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:13.973360 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:13.973360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:13.973361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:13.973502 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:13.973648 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:13.973729 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:31:13.973793 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:31:13.974186 27704 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45864
I20260928 06:31:13.974607 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27766
I20260928 06:31:13.979748 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27900
I20260928 06:31:13.984159 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28034
I20260928 06:31:13.988453 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 27672
2026-09-28T06:31:13Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1843 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:14 dist-test-slave-38pg krb5kdc[28175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:14 dist-test-slave-38pg krb5kdc[28175](info): set up 2 sockets
Sep 28 06:31:14 dist-test-slave-38pg krb5kdc[28175](info): commencing operation
krb5kdc: starting...
W20260928 06:31:15.030689 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:15.149312 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39363
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:15.230738 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:15.230997 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:15.231036 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:15.232218 28188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:15.232287 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:15.232312 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:15.232344 28188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:15.232391 28188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:15.234009 28188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:15.234285 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:15.234521 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:15.239517 28188 server_base.cc:1034] running on GCE node
W20260928 06:31:15.239423 28196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.239445 28194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.239423 28193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:15.240008 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:15.240319 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:15.241492 28188 hybrid_clock.cc:648] HybridClock initialized: now 1790577075241482 us; error 31 us; skew 500 ppm
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:15.243978 28188 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:15.244407 28188 webserver.cc:460] Webserver started at http://127.24.11.190:37607/ using document root <none> and password file <none>
I20260928 06:31:15.244632 28188 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:15.244699 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:15.244843 28188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:15.246026 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "6b1e046359bd452fad5181ed000344db"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "56194529a12f3ce2794384b1c8278a1f"
server_key_iv: "8bd36e664462fa79cdb4a9ff389686da"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.246399 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "6b1e046359bd452fad5181ed000344db"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "56194529a12f3ce2794384b1c8278a1f"
server_key_iv: "8bd36e664462fa79cdb4a9ff389686da"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.247833 28188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:15.248672 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.249002 28188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:15.249130 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "6b1e046359bd452fad5181ed000344db"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "56194529a12f3ce2794384b1c8278a1f"
server_key_iv: "8bd36e664462fa79cdb4a9ff389686da"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.249212 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:15.274606 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:15.276701 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:15.276881 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:15.281165 28188 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:39363
I20260928 06:31:15.281204 28255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:39363 every 8 connection(s)
I20260928 06:31:15.281764 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:15.282486 28256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:15.284384 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28188
I20260928 06:31:15.284498 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:15.284672 24622 external_mini_cluster.cc:1428] Setting key 7c336f038b0516c85369ae9be20da035
I20260928 06:31:15.284941 28256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db: Bootstrap starting.
I20260928 06:31:15.285714 28256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:15.286054 28256 log.cc:824] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:15.286741 28256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db: No bootstrap required, opened a new log
I20260928 06:31:15.288555 28256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } }
I20260928 06:31:15.288714 28256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:15.288762 28256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1e046359bd452fad5181ed000344db, State: Initialized, Role: FOLLOWER
I20260928 06:31:15.288910 28256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } }
I20260928 06:31:15.288990 28256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:15.289034 28256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:15.289081 28256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:15.289649 28256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } }
I20260928 06:31:15.289760 28256 leader_election.cc:302] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b1e046359bd452fad5181ed000344db; no voters:
I20260928 06:31:15.289952 28256 leader_election.cc:288] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:15.290107 28262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:15.290216 28256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:15.290236 28262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [term 1 LEADER]: Becoming Leader. State: Replica: 6b1e046359bd452fad5181ed000344db, State: Running, Role: LEADER
I20260928 06:31:15.290464 28262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } }
I20260928 06:31:15.290845 28262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b1e046359bd452fad5181ed000344db. Latest consensus state: current_term: 1 leader_uuid: "6b1e046359bd452fad5181ed000344db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } } }
I20260928 06:31:15.291003 28262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [sys.catalog]: This master's current role is: LEADER
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:15.291831 28263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b1e046359bd452fad5181ed000344db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b1e046359bd452fad5181ed000344db" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39363 } } }
I20260928 06:31:15.291977 28263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:15.292136 28270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:15.292654 28270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:15.294997 28270 catalog_manager.cc:1347] Generated new cluster ID: ea6ab65db58f4caf8cac85dcd2f43256
I20260928 06:31:15.295042 28270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:15.295799 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.285193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.24.11.190:39363)
0928 06:31:15.285402 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.285414 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.285449 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:31:15.286104 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.286114 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.286176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.286347 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.287019 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.287878 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.288656 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.289269 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.289386 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.292340 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.292343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.292343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.292356 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.294076 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.294511 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.294514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.294514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.294630 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.294840 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.294841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.294841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.295158 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.295300 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.295470 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.295600 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260928 06:31:15.300539 28270 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:15.301045 28270 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:15.308739 28270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b1e046359bd452fad5181ed000344db: Generated new TSK 0
I20260928 06:31:15.308957 28270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:15.335970 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:15.418969 28282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:15.419165 28282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:15.419188 28282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:15.420346 28282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:15.420385 28282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:15.420441 28282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:15.421991 28282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:15.422183 28282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:15.422350 28282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:15.427233 28282 server_base.cc:1034] running on GCE node
W20260928 06:31:15.427377 28288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.427301 28290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.427301 28287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:15.427675 28282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:15.427922 28282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:15.429064 28282 hybrid_clock.cc:648] HybridClock initialized: now 1790577075429054 us; error 28 us; skew 500 ppm
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:15.431650 28282 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:15.432047 28282 webserver.cc:460] Webserver started at http://127.24.11.129:45225/ using document root <none> and password file <none>
I20260928 06:31:15.432242 28282 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:15.432293 28282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:15.432389 28282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:15.433425 28282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "b8374f79fe274915a9253c83b9dac9fd"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "63b15cfbb46631b01d8f918ea78f7547"
server_key_iv: "a4d661a347d57c3e6d7bc09963c9cb73"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.433792 28282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "b8374f79fe274915a9253c83b9dac9fd"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "63b15cfbb46631b01d8f918ea78f7547"
server_key_iv: "a4d661a347d57c3e6d7bc09963c9cb73"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.435098 28282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:15.435834 28297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.436017 28282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.436095 28282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "b8374f79fe274915a9253c83b9dac9fd"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "63b15cfbb46631b01d8f918ea78f7547"
server_key_iv: "a4d661a347d57c3e6d7bc09963c9cb73"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.436149 28282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:15.452509 28282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:15.454867 28282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:15.455011 28282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:15.455217 28282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:15.455688 28282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:15.455734 28282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.455758 28282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:15.455804 28282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.461714 28282 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:42589
I20260928 06:31:15.461797 28410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:42589 every 8 connection(s)
I20260928 06:31:15.462117 28282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:15.471285 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28282
I20260928 06:31:15.471403 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:15.471640 24622 external_mini_cluster.cc:1428] Setting key 499b76d19e4c1b9a37a5bba48da55f6d
I20260928 06:31:15.472124 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.462885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:45241 (local address 127.24.11.190:39363)
0928 06:31:15.462952 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.462954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.463169 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:31:15.464174 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.464175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.464187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.464237 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.464914 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.465436 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.466486 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.466694 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.466776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.469034 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.469036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.469036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.469045 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.470526 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.470950 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.470952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.470952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.471014 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.471321 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.471322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.471323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.471502 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.471568 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.471926 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.472029 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:15.472281 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.462852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39363 (local address 127.24.11.129:45241)
0928 06:31:15.463047 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:15.463084 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:15.464057 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:15.464327 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:15.464332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:15.464608 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:15.464808 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.464812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.465560 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.465561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.466394 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.466396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.466787 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.466789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.466921 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.467466 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:15.467475 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:15.468939 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:15.470623 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.470631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.470633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.470866 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.471079 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.471080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.471081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.471235 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.471576 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:15.471584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:15.471759 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:31:15.471898 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 06:31:15.472070 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":97,"threads_started":1}
I20260928 06:31:15.472801 28411 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39363
I20260928 06:31:15.472888 28411 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:15.473079 28411 heartbeater.cc:511] Master 127.24.11.190:39363 requested a full tablet report, sending...
I20260928 06:31:15.473678 28220 ts_manager.cc:194] Registered new tserver with Master: b8374f79fe274915a9253c83b9dac9fd (127.24.11.129:42589)
I20260928 06:31:15.474360 28220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:45241
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:15.496037 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:15.577986 28416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:15.578207 28416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:15.578231 28416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:15.579385 28416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:15.579425 28416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:15.579452 28416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:15.581113 28416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:15.581316 28416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:15.581498 28416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:15.586450 28422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:15.586484 28416 server_base.cc:1034] running on GCE node
W20260928 06:31:15.586423 28424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.586452 28421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:15.586856 28416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:15.587097 28416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:15.588289 28416 hybrid_clock.cc:648] HybridClock initialized: now 1790577075588285 us; error 39 us; skew 500 ppm
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:15.591001 28416 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:15.591434 28416 webserver.cc:460] Webserver started at http://127.24.11.130:35229/ using document root <none> and password file <none>
I20260928 06:31:15.591614 28416 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:15.591655 28416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:15.591742 28416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:15.592770 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "3d110877a69d438eb3c7c6c63f0aea8b"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "ce48b6e32acf418d5f9fa1acda23d4a7"
server_key_iv: "ccf98826067b712115d6a0a1bf922875"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.593072 28416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "3d110877a69d438eb3c7c6c63f0aea8b"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "ce48b6e32acf418d5f9fa1acda23d4a7"
server_key_iv: "ccf98826067b712115d6a0a1bf922875"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.594437 28416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:15.595180 28431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.595329 28416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:15.595419 28416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "3d110877a69d438eb3c7c6c63f0aea8b"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "ce48b6e32acf418d5f9fa1acda23d4a7"
server_key_iv: "ccf98826067b712115d6a0a1bf922875"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.595475 28416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:15.617985 28416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:15.620462 28416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:15.620582 28416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:15.620831 28416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:15.621153 28416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:15.621187 28416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.621210 28416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:15.621251 28416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.628192 28416 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:41995
I20260928 06:31:15.628647 28416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:15.629463 28544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:41995 every 8 connection(s)
I20260928 06:31:15.630924 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28416
I20260928 06:31:15.631052 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:15.631228 24622 external_mini_cluster.cc:1428] Setting key e4629cc900e56ba775b58b86f009fe8d
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:15.640359 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.629494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:46461 (local address 127.24.11.190:39363)
0928 06:31:15.629564 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.629566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.630024 (+ 458us) server_negotiation.cc:407] Connection header received
0928 06:31:15.631429 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.631431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.631446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.631514 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.632445 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.633227 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.634320 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.634529 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.634612 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.637212 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.637214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.637214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.637221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.638860 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.639348 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.639350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.639350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.639412 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.639662 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.639663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.639663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.639778 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.639853 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.640181 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.640264 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:31:15.640554 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.629507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39363 (local address 127.24.11.130:46461)
0928 06:31:15.629864 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:15.629915 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:15.631283 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:15.631596 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:15.631601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:15.632033 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:15.632338 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.632343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.633347 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.633348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.634227 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.634229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.634622 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.634623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.634768 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.635307 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:15.635316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:15.637115 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:15.638956 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.638964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.638966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.639264 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.639477 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.639478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.639478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.639596 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.639842 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:15.639920 (+ 78us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:15.640046 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:31:15.640158 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:31:15.640320 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":102,"threads_started":1}
I20260928 06:31:15.640977 28545 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39363
I20260928 06:31:15.641090 28545 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:15.641271 28545 heartbeater.cc:511] Master 127.24.11.190:39363 requested a full tablet report, sending...
I20260928 06:31:15.641683 28220 ts_manager.cc:194] Registered new tserver with Master: 3d110877a69d438eb3c7c6c63f0aea8b (127.24.11.130:41995)
I20260928 06:31:15.642078 28220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:46461
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:15.664592 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:15.771893 28550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:15.772109 28550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:15.772135 28550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:15.773339 28550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:15.773386 28550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:15.773412 28550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:15.774956 28550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:15.775209 28550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:15.775405 28550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:15.780448 28558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.780452 28556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:15.780448 28555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:15.780731 28550 server_base.cc:1034] running on GCE node
I20260928 06:31:15.780968 28550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:15.781270 28550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:15.782436 28550 hybrid_clock.cc:648] HybridClock initialized: now 1790577075782416 us; error 39 us; skew 500 ppm
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:15.785175 28550 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:15.785701 28550 webserver.cc:460] Webserver started at http://127.24.11.131:34135/ using document root <none> and password file <none>
I20260928 06:31:15.785960 28550 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:15.786011 28550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:15.786136 28550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:15.787223 28550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "ec42ed2270024e4e80f531fcce59979e"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "4339e8cb808257abbbec904b9bccc77e"
server_key_iv: "76037875d2db1dc63e6c812181b007ba"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.787604 28550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "ec42ed2270024e4e80f531fcce59979e"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "4339e8cb808257abbbec904b9bccc77e"
server_key_iv: "76037875d2db1dc63e6c812181b007ba"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.789063 28550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:15.789999 28565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.790169 28550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:15.790269 28550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "ec42ed2270024e4e80f531fcce59979e"
format_stamp: "Formatted at 2026-09-28 06:31:15 on dist-test-slave-38pg"
server_key: "4339e8cb808257abbbec904b9bccc77e"
server_key_iv: "76037875d2db1dc63e6c812181b007ba"
server_key_version: "encryptionkey@0"
I20260928 06:31:15.790356 28550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:15.807780 28550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:15.810385 28550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:15.810570 28550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:15.810854 28550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:15.811239 28550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:15.811275 28550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.811298 28550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:15.811311 28550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:15.817240 28550 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37725
I20260928 06:31:15.817293 28678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37725 every 8 connection(s)
I20260928 06:31:15.817662 28550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:15.820446 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28550
I20260928 06:31:15.820556 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:15.820717 24622 external_mini_cluster.cc:1428] Setting key 6913c2e1aaa87d8191c6ba61b1e6ed54
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:15.827287 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.818429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52905 (local address 127.24.11.190:39363)
0928 06:31:15.818502 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.818504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.818688 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:31:15.819732 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.819733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.819743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.819807 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.820447 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.821146 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.822306 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.822519 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.822602 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.824719 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.824720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.824721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.824727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.826142 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.826507 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.826508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.826508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.826558 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.826757 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.826758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.826758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.826859 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.826908 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.827158 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.827220 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:31:15.827513 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.818396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39363 (local address 127.24.11.131:52905)
0928 06:31:15.818563 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:15.818606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:15.819639 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:15.819878 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:15.819882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:15.820145 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:15.820336 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.820340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.821243 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.821243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.822204 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:15.822206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.822609 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.822611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:15.822749 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.823213 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:15.823221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:15.824635 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:15.826213 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.826219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.826221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.826445 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.826611 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:15.826612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:15.826612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:15.826709 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:15.826933 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:15.826937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:15.827053 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 06:31:15.827137 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:31:15.827259 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260928 06:31:15.828033 28679 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39363
I20260928 06:31:15.828132 28679 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:15.828302 28679 heartbeater.cc:511] Master 127.24.11.190:39363 requested a full tablet report, sending...
I20260928 06:31:15.828681 28220 ts_manager.cc:194] Registered new tserver with Master: ec42ed2270024e4e80f531fcce59979e (127.24.11.131:37725)
I20260928 06:31:15.829031 28220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:52905
I20260928 06:31:15.832772 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:31:15.856640 28686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.848256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.24.11.129:42589)
0928 06:31:15.848454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.848459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.848468 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:15.848493 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.848495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.848508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.848694 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.849283 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.850079 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.850837 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.851167 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.851241 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.853809 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.853820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.853824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.853834 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.855396 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.855791 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.855796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.855796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.855867 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.856067 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.856068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.856069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.856220 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.856390 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.856437 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.856520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:15.864615 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:15.857939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.24.11.190:39363)
0928 06:31:15.857980 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:15.857982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:15.858005 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:31:15.858057 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:15.858058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:15.858067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:15.858116 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:15.858616 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.859113 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.859794 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:15.859985 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:15.860045 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:15.862199 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:15.862200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:15.862201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:15.862206 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:15.863375 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.863697 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.863697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.863697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.863747 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:15.863945 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:15.863945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:15.863946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:15.864084 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:15.864162 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:15.864477 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 06:31:15.864550 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:31:15.864909 28220 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43874
I20260928 06:31:15.865296 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28282
I20260928 06:31:15.870107 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28416
I20260928 06:31:15.874271 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28550
I20260928 06:31:15.878289 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28188
2026-09-28T06:31:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1889 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28691](info): set up 2 sockets
Sep 28 06:31:15 dist-test-slave-38pg krb5kdc[28691](info): commencing operation
krb5kdc: starting...
W20260928 06:31:16.968510 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 06:31:17.180294 24622 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.212s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 06:31:17.458009 24622 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.278s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 06:31:17.582969 24622 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.125s user 0.002s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/test-user.keytab.
W20260928 06:31:17.873907 24622 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.291s user 0.001s sys 0.000s
Sep 28 06:31:17 dist-test-slave-38pg krb5kdc[28691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:17.918751 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35769
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:17.998896 28704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:17.999151 28704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:17.999193 28704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.000437 28704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:18.000507 28704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.000531 28704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.000552 28704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:18.000582 28704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:18.002156 28704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.002447 28704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.002734 28704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.007676 28710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.007644 28709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.007738 28704 server_base.cc:1034] running on GCE node
W20260928 06:31:18.007644 28712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.008155 28704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.008457 28704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.009603 28704 hybrid_clock.cc:648] HybridClock initialized: now 1790577078009598 us; error 37 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.012233 28704 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:18.012648 28704 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.012720 28704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.012914 28704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.014127 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "1d8d07d8d21e434fb6974abcfcddacd9"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "57ba0419ffcfc7b84bac82507fa579e1"
server_key_iv: "f71ff479d6707d20d559469fd1146aad"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.014501 28704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "1d8d07d8d21e434fb6974abcfcddacd9"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "57ba0419ffcfc7b84bac82507fa579e1"
server_key_iv: "f71ff479d6707d20d559469fd1146aad"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.015882 28704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:18.016615 28718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.016837 28704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.016948 28704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "1d8d07d8d21e434fb6974abcfcddacd9"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "57ba0419ffcfc7b84bac82507fa579e1"
server_key_iv: "f71ff479d6707d20d559469fd1146aad"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.017040 28704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.025466 28704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.027537 28704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.027716 28704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.031754 28704 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:35769
I20260928 06:31:18.031780 28770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:35769 every 8 connection(s)
I20260928 06:31:18.032079 28704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:18.033046 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:18.033314 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28704
I20260928 06:31:18.033423 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:18.033632 24622 external_mini_cluster.cc:1428] Setting key 7d902e33d5e5ed926186a87a558f53cb
I20260928 06:31:18.036484 28771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: Bootstrap starting.
I20260928 06:31:18.037272 28771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:18.037595 28771 log.cc:824] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.038321 28771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: No bootstrap required, opened a new log
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:18.040495 28771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } }
I20260928 06:31:18.040655 28771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:18.040705 28771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8d07d8d21e434fb6974abcfcddacd9, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.040854 28771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } }
I20260928 06:31:18.040951 28771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:18.040992 28771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:18.041040 28771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:18.042631 28771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } }
I20260928 06:31:18.042725 28771 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8d07d8d21e434fb6974abcfcddacd9; no voters:
I20260928 06:31:18.042999 28771 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:18.043026 28776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:18.043138 28776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8d07d8d21e434fb6974abcfcddacd9, State: Running, Role: LEADER
I20260928 06:31:18.043292 28771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:18.043329 28776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } }
I20260928 06:31:18.043825 28777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } } }
I20260928 06:31:18.044086 28777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:18.044010 28778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8d07d8d21e434fb6974abcfcddacd9. Latest consensus state: current_term: 1 leader_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8d07d8d21e434fb6974abcfcddacd9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35769 } } }
I20260928 06:31:18.044356 28778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:18.044564 28785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:18.044950 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.034161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.24.11.190:35769)
0928 06:31:18.034341 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.034348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.034373 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:18.034837 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.034844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.034894 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.035011 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.035794 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.036790 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.037898 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.038420 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.038529 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.041188 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.041191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.041192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:18.041202 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.042865 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.043290 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.043293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.043293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.043438 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.043613 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.043614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.043614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.043940 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.044097 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.044575 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.044714 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 06:31:18.045042 28785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:18.046486 28785 catalog_manager.cc:1347] Generated new cluster ID: 9e55d8ed535a491a8eb8785459025136
I20260928 06:31:18.046542 28785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:18.062958 28785 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:18.063750 28785 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:18.075798 28785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: Generated new TSK 0
I20260928 06:31:18.076038 28785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:18.104475 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:18.185129 28797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.185375 28797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.185412 28797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.186729 28797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.186790 28797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.186836 28797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:18.188359 28797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.188627 28797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.188866 28797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.193794 28803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.193789 28805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.193816 28802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.194188 28797 server_base.cc:1034] running on GCE node
I20260928 06:31:18.194386 28797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.194581 28797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.195720 28797 hybrid_clock.cc:648] HybridClock initialized: now 1790577078195701 us; error 33 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.198351 28797 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:18.198751 28797 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.198823 28797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.198979 28797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.200065 28797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "616dd01467d74e9d99bd197c498dd6d1"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f77e787520db19b367033c09e94c7289"
server_key_iv: "825b8d4d45c20437a9cf4e18c1030535"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.200445 28797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "616dd01467d74e9d99bd197c498dd6d1"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f77e787520db19b367033c09e94c7289"
server_key_iv: "825b8d4d45c20437a9cf4e18c1030535"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.201866 28797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:18.202615 28811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.202860 28797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.202979 28797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "616dd01467d74e9d99bd197c498dd6d1"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f77e787520db19b367033c09e94c7289"
server_key_iv: "825b8d4d45c20437a9cf4e18c1030535"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.203063 28797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.216364 28797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.218369 28797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.218538 28797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.218791 28797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.219102 28797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:18.219156 28797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.219201 28797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:18.219225 28797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.225317 28797 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:42521
I20260928 06:31:18.225356 28924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:42521 every 8 connection(s)
I20260928 06:31:18.225683 28797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:18.229792 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28797
I20260928 06:31:18.229915 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:18.230080 24622 external_mini_cluster.cc:1428] Setting key dd54525f0af133994d291623c36658a3
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:18.236164 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.226384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:39663 (local address 127.24.11.190:35769)
0928 06:31:18.226431 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.226433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.226645 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:31:18.227112 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.227113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.227123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.227185 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.228153 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.228637 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.229717 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.230107 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.230187 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.232645 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.232646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.232647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:18.232655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.234597 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.235062 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.235064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.235065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.235134 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.235391 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.235392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.235393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.235550 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.235630 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.235926 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.236029 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:31:18.236208 28927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.226350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35769 (local address 127.24.11.129:39663)
0928 06:31:18.226515 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.226557 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.227030 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.227252 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.227256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.227520 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:18.228068 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.228073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.228718 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.228719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.229629 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.229632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.230222 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.230223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.230366 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.230963 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:18.230973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:18.232551 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:18.234685 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.234692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.234693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.234981 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.235209 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.235210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.235210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.235322 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.235627 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:18.235631 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:18.235786 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.235882 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.236015 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 06:31:18.236845 28925 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35769
I20260928 06:31:18.236958 28925 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:18.237133 28925 heartbeater.cc:511] Master 127.24.11.190:35769 requested a full tablet report, sending...
I20260928 06:31:18.237746 28735 ts_manager.cc:194] Registered new tserver with Master: 616dd01467d74e9d99bd197c498dd6d1 (127.24.11.129:42521)
I20260928 06:31:18.238363 28735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:39663
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:18.257301 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:18.338035 28930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.338238 28930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.338261 28930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.339370 28930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.339409 28930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.339439 28930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:18.340867 28930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.341061 28930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.341233 28930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.346329 28936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.346336 28938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.346347 28935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.346697 28930 server_base.cc:1034] running on GCE node
I20260928 06:31:18.346948 28930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.347198 28930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.348357 28930 hybrid_clock.cc:648] HybridClock initialized: now 1790577078348345 us; error 31 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.350958 28930 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:18.351257 28930 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.351303 28930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.351400 28930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.352454 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "e9570ef35e504c5d99c9baee9927f620"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f68cfc0db915991cdcb8bf01ef612aff"
server_key_iv: "301238e3bf60f495be5bb36ed4d4267d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.352772 28930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "e9570ef35e504c5d99c9baee9927f620"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f68cfc0db915991cdcb8bf01ef612aff"
server_key_iv: "301238e3bf60f495be5bb36ed4d4267d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.354208 28930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:31:18.355003 28944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.355216 28930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.355290 28930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "e9570ef35e504c5d99c9baee9927f620"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "f68cfc0db915991cdcb8bf01ef612aff"
server_key_iv: "301238e3bf60f495be5bb36ed4d4267d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.355381 28930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.388353 28930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.390422 28930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.390566 28930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.390828 28930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.391150 28930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:18.391183 28930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.391206 28930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:18.391250 28930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.397064 28930 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:45401
I20260928 06:31:18.397132 29057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:45401 every 8 connection(s)
I20260928 06:31:18.397408 28930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:18.402402 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 28930
I20260928 06:31:18.402524 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:18.402702 24622 external_mini_cluster.cc:1428] Setting key dca6d627933fb336f692952bc54b00d5
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:18.407009 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.398185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:40493 (local address 127.24.11.190:35769)
0928 06:31:18.398283 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.398286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.398490 (+ 204us) server_negotiation.cc:407] Connection header received
0928 06:31:18.398944 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.398945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.398956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.399011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.399897 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.400399 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.401355 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.401564 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.401675 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.403844 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.403846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.403846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:18.403854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.405334 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.405910 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.405911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.405912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.405964 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.406256 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.406256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.406256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.406358 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.406408 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.406812 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.406934 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:31:18.407194 29060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.398136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35769 (local address 127.24.11.130:40493)
0928 06:31:18.398348 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.398403 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.398856 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.399063 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.399066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.399318 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:18.399813 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.399818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.400482 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.400482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.401255 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.401257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.401679 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.401680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.401778 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.402221 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:18.402229 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:18.403727 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:18.405421 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.405428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.405430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.405788 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.406030 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.406031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.406032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.406194 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.406432 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:18.406437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:18.406601 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.406714 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.406893 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":115,"threads_started":1}
I20260928 06:31:18.407902 29058 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35769
I20260928 06:31:18.408010 29058 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:18.408175 29058 heartbeater.cc:511] Master 127.24.11.190:35769 requested a full tablet report, sending...
I20260928 06:31:18.408563 28735 ts_manager.cc:194] Registered new tserver with Master: e9570ef35e504c5d99c9baee9927f620 (127.24.11.130:45401)
I20260928 06:31:18.408963 28735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:40493
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:18.428983 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:18.510836 29063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.511042 29063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.511067 29063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.512208 29063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.512252 29063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.512312 29063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:18.513893 29063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.514127 29063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.514379 29063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.519423 29069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.519448 29071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.519551 29063 server_base.cc:1034] running on GCE node
W20260928 06:31:18.519654 29068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.519892 29063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.520166 29063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.521315 29063 hybrid_clock.cc:648] HybridClock initialized: now 1790577078521292 us; error 37 us; skew 500 ppm
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:18.523900 29063 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:18.524189 29063 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.524231 29063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.524319 29063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.525327 29063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "d625a24b92bf477485e197b972370589"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "172712d06ad55a9b3d2596f2d2da5bb4"
server_key_iv: "f917bb28d897f85b901c70ea92c387e7"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.525717 29063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "d625a24b92bf477485e197b972370589"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "172712d06ad55a9b3d2596f2d2da5bb4"
server_key_iv: "f917bb28d897f85b901c70ea92c387e7"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.526932 29063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:18.527580 29077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.527776 29063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:18.527895 29063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "d625a24b92bf477485e197b972370589"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "172712d06ad55a9b3d2596f2d2da5bb4"
server_key_iv: "f917bb28d897f85b901c70ea92c387e7"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.527972 29063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.537665 29063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.539525 29063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.539676 29063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.539901 29063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.540200 29063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:18.540232 29063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.540274 29063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:18.540301 29063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.546463 29063 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:35855
I20260928 06:31:18.546546 29190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:35855 every 8 connection(s)
I20260928 06:31:18.546780 29063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:31:18 dist-test-slave-38pg krb5kdc[28691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:18.553958 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29063
I20260928 06:31:18.554093 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:18.554277 24622 external_mini_cluster.cc:1428] Setting key 3d0d38fa40ff70b1170fbcd8f8f0719e
I20260928 06:31:18.556030 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.547469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:45581 (local address 127.24.11.190:35769)
0928 06:31:18.547516 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.547518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.547735 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:31:18.548197 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.548198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.548207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.548258 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.549103 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.549660 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.550688 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.550940 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.551013 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.552901 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.552903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.552903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:18.552923 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.554535 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.555052 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.555053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.555053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.555117 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.555375 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.555376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.555376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.555503 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.555581 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.555853 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.555941 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:31:18.556224 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.547451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35769 (local address 127.24.11.131:45581)
0928 06:31:18.547618 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.547651 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.548117 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.548309 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.548313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.548554 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:18.549028 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.549032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.549732 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.549732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.550525 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.550527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.551023 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.551024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.551119 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.551535 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:18.551543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:18.552820 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:18.554624 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.554629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.554631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.554976 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.555178 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:18.555180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:18.555180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:18.555302 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:18.555573 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:18.555579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:18.555719 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.555831 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.555989 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260928 06:31:18.556663 29191 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35769
I20260928 06:31:18.556742 29191 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:18.556905 29191 heartbeater.cc:511] Master 127.24.11.190:35769 requested a full tablet report, sending...
I20260928 06:31:18.557255 28735 ts_manager.cc:194] Registered new tserver with Master: d625a24b92bf477485e197b972370589 (127.24.11.131:35855)
I20260928 06:31:18.557659 28735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:45581
I20260928 06:31:18.566119 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:18.573028 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.567355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.24.11.190:35769)
0928 06:31:18.567416 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.567419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.567531 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:31:18.567571 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.567571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.567580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.567631 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.568131 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.568590 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.569301 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.569474 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.569564 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.570660 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.570661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.570661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:18.570668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.572068 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.572357 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.572357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.572358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.572408 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:18.572555 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:18.572555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:18.572555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:18.572684 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.572752 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.572850 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.572906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:31:18.574641 28735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:18.576226 28735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:18.582688 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:42521 (local address 127.0.0.1:53212)
0928 06:31:18.578663 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.578709 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.578777 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.579534 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.579535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.579537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:18.579585 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.579587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580755 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.580756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.581722 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.581725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.582354 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.582355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.582456 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.582460 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.582526 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.582591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260928 06:31:18.582710 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.24.11.130:45401)
0928 06:31:18.578832 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.578837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.578860 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:31:18.578900 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.578902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.578917 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.579007 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:18.579221 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580401 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.581730 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.582261 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.582347 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.582408 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.582561 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.582606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260928 06:31:18.582932 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.24.11.129:42521)
0928 06:31:18.579384 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.579388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.579397 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:18.579432 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.579434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.579448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.579521 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:18.579650 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580663 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.581794 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.582261 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.582362 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.582422 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.582809 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.582855 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":79,"threads_started":1}
I20260928 06:31:18.582690 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:45401 (local address 127.0.0.1:56264)
0928 06:31:18.578780 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.578792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.578842 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.579032 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.579036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.579038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:18.579106 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.579108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580520 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.580521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.581648 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.581651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.582360 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.582361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.582441 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.582454 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.582528 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.582599 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260928 06:31:18.583534 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35855 (local address 127.0.0.1:43438)
0928 06:31:18.578918 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.578927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.578962 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.579257 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.579257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.579260 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:18.579308 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.579310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580302 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.580303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.581648 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.581651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.583269 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.583270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.583332 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.583336 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.583384 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.583441 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":7,"thread_start_us":59,"threads_started":1}
I20260928 06:31:18.583688 28859 tablet_service.cc:1467] Processing CreateTablet for tablet e3da64295a934b2cb4da81bbceb888a4 (DEFAULT_TABLE table=test-table [id=858f5135916747198b28bd4f3c3e7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:18.584064 28859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3da64295a934b2cb4da81bbceb888a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:18.584061 28992 tablet_service.cc:1467] Processing CreateTablet for tablet e3da64295a934b2cb4da81bbceb888a4 (DEFAULT_TABLE table=test-table [id=858f5135916747198b28bd4f3c3e7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:18.584307 28992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3da64295a934b2cb4da81bbceb888a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:18.586663 29208 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: Bootstrap starting.
I20260928 06:31:18.587102 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.578892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.24.11.131:35855)
0928 06:31:18.579100 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.579104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.579112 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:18.579142 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.579143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.579154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.579248 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:18.579393 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.580202 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.582448 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.583188 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.583256 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.583306 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.586958 (+ 3652us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.587021 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
I20260928 06:31:18.587350 29208 tablet_bootstrap.cc:654] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:18.587666 29209 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Bootstrap starting.
I20260928 06:31:18.587680 29125 tablet_service.cc:1467] Processing CreateTablet for tablet e3da64295a934b2cb4da81bbceb888a4 (DEFAULT_TABLE table=test-table [id=858f5135916747198b28bd4f3c3e7def]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:18.588014 29125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3da64295a934b2cb4da81bbceb888a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:18.587983 29208 log.cc:824] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.588356 29209 tablet_bootstrap.cc:654] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:18.588773 29208 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: No bootstrap required, opened a new log
I20260928 06:31:18.588851 29208 ts_tablet_manager.cc:1397] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:18.588868 29209 log.cc:824] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.589613 29209 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: No bootstrap required, opened a new log
I20260928 06:31:18.589699 29209 ts_tablet_manager.cc:1397] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:18.591048 29208 raft_consensus.cc:348] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.591310 29208 raft_consensus.cc:374] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:18.591341 29212 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Bootstrap starting.
I20260928 06:31:18.591436 29208 raft_consensus.cc:729] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 616dd01467d74e9d99bd197c498dd6d1, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.591182 29209 raft_consensus.cc:348] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.591663 29209 raft_consensus.cc:374] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:18.591646 29208 consensus_queue.cc:260] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.591733 29209 raft_consensus.cc:729] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9570ef35e504c5d99c9baee9927f620, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.591854 29209 consensus_queue.cc:260] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.591950 29208 ts_tablet_manager.cc:1428] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:18.591989 29212 tablet_bootstrap.cc:654] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:18.592082 28925 heartbeater.cc:503] Master 127.24.11.190:35769 was elected leader, sending a full tablet report...
I20260928 06:31:18.592170 29209 ts_tablet_manager.cc:1428] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:18.592294 29058 heartbeater.cc:503] Master 127.24.11.190:35769 was elected leader, sending a full tablet report...
I20260928 06:31:18.592628 29212 log.cc:824] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.593493 29212 tablet_bootstrap.cc:492] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: No bootstrap required, opened a new log
I20260928 06:31:18.593598 29212 ts_tablet_manager.cc:1397] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:18.594869 29212 raft_consensus.cc:348] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.594988 29212 raft_consensus.cc:374] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:18.595009 29212 raft_consensus.cc:729] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d625a24b92bf477485e197b972370589, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.595109 29212 consensus_queue.cc:260] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.595350 29212 ts_tablet_manager.cc:1428] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:31:18.595381 29191 heartbeater.cc:503] Master 127.24.11.190:35769 was elected leader, sending a full tablet report...
W20260928 06:31:18.648005 29059 tablet.cc:2369] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:18.654656 29216 raft_consensus.cc:482] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:18.654799 29216 raft_consensus.cc:504] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.655251 29216 leader_election.cc:288] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 616dd01467d74e9d99bd197c498dd6d1 (127.24.11.129:42521), e9570ef35e504c5d99c9baee9927f620 (127.24.11.130:45401)
I20260928 06:31:18.659272 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.655525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:60583 (local address 127.24.11.130:45401)
0928 06:31:18.655567 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.655569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.655722 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:31:18.655775 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.655776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.655783 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.655854 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:18.655998 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.656677 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.658354 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.658911 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.658986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.658996 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.659162 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.659202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:31:18.659272 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.655502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:58635 (local address 127.24.11.129:42521)
0928 06:31:18.655550 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.655552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.655561 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:18.655590 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.655590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.655600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.655665 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:18.655875 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.656655 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.658292 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.658891 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.659003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.659012 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.659162 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.659202 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:18.659287 29193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.655416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:42521 (local address 127.24.11.131:58635)
0928 06:31:18.655475 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.655486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.655530 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.655692 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.655693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.655695 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:18.655771 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.655773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.656762 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.656763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.658196 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.658199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.658993 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.658995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.659084 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.659087 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.659139 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.659187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":116}
I20260928 06:31:18.659296 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.655451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:45401 (local address 127.24.11.131:60583)
0928 06:31:18.655665 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.655674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.655724 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.655862 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.655862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.655864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:18.655929 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.655930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.656762 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.656763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.658196 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:18.658199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.658993 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.658995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:18.659074 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.659078 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.659139 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.659187 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":66,"thread_start_us":125,"threads_started":1}
I20260928 06:31:18.659652 28879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3da64295a934b2cb4da81bbceb888a4" candidate_uuid: "d625a24b92bf477485e197b972370589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616dd01467d74e9d99bd197c498dd6d1" is_pre_election: true
I20260928 06:31:18.659777 29012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3da64295a934b2cb4da81bbceb888a4" candidate_uuid: "d625a24b92bf477485e197b972370589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9570ef35e504c5d99c9baee9927f620" is_pre_election: true
I20260928 06:31:18.659819 28879 raft_consensus.cc:2454] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d625a24b92bf477485e197b972370589 in term 0.
I20260928 06:31:18.659891 29012 raft_consensus.cc:2454] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d625a24b92bf477485e197b972370589 in term 0.
I20260928 06:31:18.660094 29078 leader_election.cc:302] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616dd01467d74e9d99bd197c498dd6d1, d625a24b92bf477485e197b972370589; no voters:
I20260928 06:31:18.660259 29216 raft_consensus.cc:2781] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:18.660367 29216 raft_consensus.cc:482] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:18.660413 29216 raft_consensus.cc:3037] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:18.661208 29216 raft_consensus.cc:504] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.661377 29216 leader_election.cc:288] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [CANDIDATE]: Term 1 election: Requested vote from peers 616dd01467d74e9d99bd197c498dd6d1 (127.24.11.129:42521), e9570ef35e504c5d99c9baee9927f620 (127.24.11.130:45401)
I20260928 06:31:18.661573 28879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3da64295a934b2cb4da81bbceb888a4" candidate_uuid: "d625a24b92bf477485e197b972370589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "616dd01467d74e9d99bd197c498dd6d1"
I20260928 06:31:18.661577 29012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3da64295a934b2cb4da81bbceb888a4" candidate_uuid: "d625a24b92bf477485e197b972370589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9570ef35e504c5d99c9baee9927f620"
I20260928 06:31:18.661657 29012 raft_consensus.cc:3037] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:18.661659 28879 raft_consensus.cc:3037] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:18.662554 29012 raft_consensus.cc:2454] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d625a24b92bf477485e197b972370589 in term 1.
I20260928 06:31:18.662554 28879 raft_consensus.cc:2454] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d625a24b92bf477485e197b972370589 in term 1.
I20260928 06:31:18.662766 29078 leader_election.cc:302] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 616dd01467d74e9d99bd197c498dd6d1, d625a24b92bf477485e197b972370589; no voters:
I20260928 06:31:18.662885 29216 raft_consensus.cc:2781] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:18.663051 29216 raft_consensus.cc:686] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 1 LEADER]: Becoming Leader. State: Replica: d625a24b92bf477485e197b972370589, State: Running, Role: LEADER
I20260928 06:31:18.663160 29216 consensus_queue.cc:237] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } }
I20260928 06:31:18.663995 28734 catalog_manager.cc:5495] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 reported cstate change: term changed from 0 to 1, leader changed from <none> to d625a24b92bf477485e197b972370589 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "d625a24b92bf477485e197b972370589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d625a24b92bf477485e197b972370589" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35855 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:18.675760 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.673505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.24.11.131:35855)
0928 06:31:18.673589 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.673591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.673620 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:31:18.673649 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.673649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.673667 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.673721 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:18.673876 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.674334 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.675116 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:18.675289 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:18.675338 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:18.675468 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:18.675563 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:18.675625 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.675666 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.675684 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:31:18.677583 28879 raft_consensus.cc:1261] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 1 FOLLOWER]: Refusing update from remote peer d625a24b92bf477485e197b972370589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:18.677627 29012 raft_consensus.cc:1261] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 1 FOLLOWER]: Refusing update from remote peer d625a24b92bf477485e197b972370589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:18.677929 29218 consensus_queue.cc:1035] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "616dd01467d74e9d99bd197c498dd6d1" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:18.678064 29216 consensus_queue.cc:1035] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9570ef35e504c5d99c9baee9927f620" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:18.680917 29221 mvcc.cc:205] Tried to move back new op lower bound from 7334203714260434944 to 7334203714204934144. Current Snapshot: MvccSnapshot[applied={T|T < 7334203714260434944 or (T in {7334203714260434944})}]
I20260928 06:31:18.681020 29222 mvcc.cc:205] Tried to move back new op lower bound from 7334203714260434944 to 7334203714204934144. Current Snapshot: MvccSnapshot[applied={T|T < 7334203714260434944}]
I20260928 06:31:18.682331 29223 mvcc.cc:205] Tried to move back new op lower bound from 7334203714260434944 to 7334203714204934144. Current Snapshot: MvccSnapshot[applied={T|T < 7334203714260434944 or (T in {7334203714260434944})}]
I20260928 06:31:18.682614 28733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:18.682703 28733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:18.683462 28733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d8d07d8d21e434fb6974abcfcddacd9: Sending DeleteTablet for 3 replicas of tablet e3da64295a934b2cb4da81bbceb888a4
I20260928 06:31:18.683761 28859 tablet_service.cc:1514] Processing DeleteTablet for tablet e3da64295a934b2cb4da81bbceb888a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:18 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:53212
I20260928 06:31:18.683784 29125 tablet_service.cc:1514] Processing DeleteTablet for tablet e3da64295a934b2cb4da81bbceb888a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:18 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43438
I20260928 06:31:18.683799 28992 tablet_service.cc:1514] Processing DeleteTablet for tablet e3da64295a934b2cb4da81bbceb888a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:18 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56264
I20260928 06:31:18.684031 29232 tablet_replica.cc:330] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: stopping tablet replica
I20260928 06:31:18.684103 29230 tablet_replica.cc:330] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1: stopping tablet replica
I20260928 06:31:18.684193 29232 raft_consensus.cc:2229] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:18.684234 29230 raft_consensus.cc:2229] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:18.684329 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28797
I20260928 06:31:18.684360 29230 raft_consensus.cc:2258] T e3da64295a934b2cb4da81bbceb888a4 P 616dd01467d74e9d99bd197c498dd6d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:18.684469 29232 raft_consensus.cc:2258] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:18.685021 29231 tablet_replica.cc:330] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: stopping tablet replica
I20260928 06:31:18.685111 29231 raft_consensus.cc:2229] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:18.685220 29231 raft_consensus.cc:2258] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:18.685684 29231 ts_tablet_manager.cc:1905] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:18.687659 29231 ts_tablet_manager.cc:1918] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:18.687786 29231 log.cc:1196] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/e3da64295a934b2cb4da81bbceb888a4
I20260928 06:31:18.688086 29231 ts_tablet_manager.cc:1939] T e3da64295a934b2cb4da81bbceb888a4 P d625a24b92bf477485e197b972370589: Deleting consensus metadata
I20260928 06:31:18.688488 28720 catalog_manager.cc:4841] TS d625a24b92bf477485e197b972370589 (127.24.11.131:35855): tablet e3da64295a934b2cb4da81bbceb888a4 (table test-table [id=858f5135916747198b28bd4f3c3e7def]) successfully deleted
I20260928 06:31:18.690197 29232 ts_tablet_manager.cc:1905] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:18.691795 28719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:42521 (error 108)
I20260928 06:31:18.691912 29232 ts_tablet_manager.cc:1918] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:18.691985 29232 log.cc:1196] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/e3da64295a934b2cb4da81bbceb888a4
I20260928 06:31:18.692302 29232 ts_tablet_manager.cc:1939] T e3da64295a934b2cb4da81bbceb888a4 P e9570ef35e504c5d99c9baee9927f620: Deleting consensus metadata
I20260928 06:31:18.692349 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28930
I20260928 06:31:18.693395 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.693238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:42521 (local address 127.0.0.1:53224)
0928 06:31:18.693311 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.693359 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:42521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 06:31:18.693554 28719 catalog_manager.cc:4584] TS 616dd01467d74e9d99bd197c498dd6d1 (127.24.11.129:42521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3da64295a934b2cb4da81bbceb888a4: Network error: Client connection negotiation failed: client connection to 127.24.11.129:42521: connect: Connection refused (error 111)
I20260928 06:31:18.698135 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29063
I20260928 06:31:18.698292 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.698184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:45401 (local address 127.0.0.1:56270)
0928 06:31:18.698234 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.698258 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:45401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:31:18.703153 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 28704
2026-09-28T06:31:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2824 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:31:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:18Z Disabled control of system clock
I20260928 06:31:18.730013 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34007
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:31:18.811617 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.811841 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.811865 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.812973 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:18.813009 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.813022 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.813035 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:18.813046 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:18.814450 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:34007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.814661 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.814919 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.820181 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.820216 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.820230 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.820289 29237 server_base.cc:1034] running on GCE node
I20260928 06:31:18.820667 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.820926 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.822062 29237 hybrid_clock.cc:648] HybridClock initialized: now 1790577078822041 us; error 41 us; skew 500 ppm
I20260928 06:31:18.823280 29237 webserver.cc:460] Webserver started at http://127.24.11.190:43807/ using document root <none> and password file <none>
I20260928 06:31:18.823477 29237 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.823519 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.823611 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.824657 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "d2daeb31e32747cf8bfd2f451c7d621a"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "6b5d09ce9018a819c9d5d8e9c501827e"
server_key_iv: "50f04bc5a1a16fe8387ac76b526e073e"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.824976 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "d2daeb31e32747cf8bfd2f451c7d621a"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "6b5d09ce9018a819c9d5d8e9c501827e"
server_key_iv: "50f04bc5a1a16fe8387ac76b526e073e"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.826334 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:18.827068 29251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.827289 29237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.827392 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "d2daeb31e32747cf8bfd2f451c7d621a"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "6b5d09ce9018a819c9d5d8e9c501827e"
server_key_iv: "50f04bc5a1a16fe8387ac76b526e073e"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.827481 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.830921 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.831200 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.831353 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.835327 29237 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:34007
I20260928 06:31:18.835399 29303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:34007 every 8 connection(s)
I20260928 06:31:18.835747 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:18.836447 29304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:18.839381 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: Bootstrap starting.
I20260928 06:31:18.839993 29304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:18.840297 29304 log.cc:824] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:18.842110 29304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: No bootstrap required, opened a new log
I20260928 06:31:18.843766 29304 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } }
I20260928 06:31:18.843923 29304 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:18.843947 29304 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2daeb31e32747cf8bfd2f451c7d621a, State: Initialized, Role: FOLLOWER
I20260928 06:31:18.844135 29304 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } }
I20260928 06:31:18.844216 29304 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:18.844238 29304 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:18.844304 29304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:18.844329 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29237
I20260928 06:31:18.844463 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:18.844637 24622 external_mini_cluster.cc:1428] Setting key 417723e4ba328233e3fff2c3ef2ba854
I20260928 06:31:18.845043 29304 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } }
I20260928 06:31:18.845153 29304 leader_election.cc:302] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2daeb31e32747cf8bfd2f451c7d621a; no voters:
I20260928 06:31:18.845332 29304 leader_election.cc:288] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:18.845386 29309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:18.845561 29309 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [term 1 LEADER]: Becoming Leader. State: Replica: d2daeb31e32747cf8bfd2f451c7d621a, State: Running, Role: LEADER
I20260928 06:31:18.845691 29304 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:18.845706 29309 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } }
I20260928 06:31:18.846556 29311 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2daeb31e32747cf8bfd2f451c7d621a. Latest consensus state: current_term: 1 leader_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } } }
I20260928 06:31:18.846654 29311 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:18.846884 29310 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2daeb31e32747cf8bfd2f451c7d621a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34007 } } }
I20260928 06:31:18.847020 29310 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:18.847489 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.845178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.24.11.190:34007)
0928 06:31:18.845399 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.845410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.845447 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:31:18.845602 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.845611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.845671 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.846102 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.846753 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.846756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.846757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:18.846760 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.846911 (+ 151us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:18.846932 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.847183 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.847218 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.847289 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 06:31:18.847682 29315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:18.848268 29315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:18.849947 29315 catalog_manager.cc:1347] Generated new cluster ID: eb48aeb7a92243268d2bd8f0fcad1caf
I20260928 06:31:18.850008 29315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:18.857329 29315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:18.857518 29315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:18.861204 29315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: Generated new TSK 0
I20260928 06:31:18.861406 29315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:18.869983 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--builtin_ntp_servers=127.24.11.148:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:31:18.951709 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:18.951962 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:18.952004 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:18.953209 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:18.953272 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:18.953318 29328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:18.954752 29328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:18.955027 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:18.955330 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:18.961046 29333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:18.961035 29334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.961262 29328 server_base.cc:1034] running on GCE node
W20260928 06:31:18.961061 29336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:18.961617 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:18.961890 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:18.963081 29328 hybrid_clock.cc:648] HybridClock initialized: now 1790577078963066 us; error 30 us; skew 500 ppm
I20260928 06:31:18.964432 29328 webserver.cc:460] Webserver started at http://127.24.11.129:35559/ using document root <none> and password file <none>
I20260928 06:31:18.964689 29328 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:18.964768 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:18.964927 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:18.966147 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "883ca521f2aa4ed4851446b9e01636c5"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "b30d1f53ed521619d6a6987257a60f38"
server_key_iv: "abfb898a5dec03edecb509f1db190d6d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.966522 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "883ca521f2aa4ed4851446b9e01636c5"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "b30d1f53ed521619d6a6987257a60f38"
server_key_iv: "abfb898a5dec03edecb509f1db190d6d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.967923 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:18.968720 29342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.968946 29328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:18.969043 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "883ca521f2aa4ed4851446b9e01636c5"
format_stamp: "Formatted at 2026-09-28 06:31:18 on dist-test-slave-38pg"
server_key: "b30d1f53ed521619d6a6987257a60f38"
server_key_iv: "abfb898a5dec03edecb509f1db190d6d"
server_key_version: "encryptionkey@0"
I20260928 06:31:18.969111 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:18.973773 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:18.974138 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:18.974296 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:18.974541 29328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:18.974915 29328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:18.974960 29328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.974987 29328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:18.975003 29328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:18.981256 29328 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:45429
I20260928 06:31:18.981369 29455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:45429 every 8 connection(s)
I20260928 06:31:18.981647 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:18.984504 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.982357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58267 (local address 127.24.11.190:34007)
0928 06:31:18.982447 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:18.982450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:18.982662 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:31:18.983138 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:18.983139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:18.983145 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:18.983199 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:18.984309 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:18.984311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:18.984311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:18.984315 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:18.984331 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:18.984335 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:18.984384 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:18.984439 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:31:18.984451 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:31:18.984715 29458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:18.982333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:34007 (local address 127.24.11.129:58267)
0928 06:31:18.982540 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:18.982574 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:18.983045 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:18.983273 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:18.983277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:18.983285 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:18.984155 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:18.984156 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:18.984185 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:18.984214 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:18.984382 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:18.984384 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:18.984386 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:18.984438 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:31:18.984486 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260928 06:31:18.984951 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29328
I20260928 06:31:18.985051 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:18.985239 24622 external_mini_cluster.cc:1428] Setting key 99273579c7783c33fc8cb2587d8c2512
I20260928 06:31:18.985273 29456 heartbeater.cc:347] Connected to a master server at 127.24.11.190:34007
I20260928 06:31:18.985378 29456 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:18.985569 29456 heartbeater.cc:511] Master 127.24.11.190:34007 requested a full tablet report, sending...
I20260928 06:31:18.986102 29268 ts_manager.cc:194] Registered new tserver with Master: 883ca521f2aa4ed4851446b9e01636c5 (127.24.11.129:45429)
I20260928 06:31:18.986166 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--builtin_ntp_servers=127.24.11.148:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:31:19.071966 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:19.072297 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:19.072377 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:19.073972 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:19.074041 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:19.074086 29459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:19.075568 29459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:19.075851 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:19.076169 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:19.081238 29465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.081229 29467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.081555 29459 server_base.cc:1034] running on GCE node
W20260928 06:31:19.081699 29464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.081916 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:19.082192 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:19.083386 29459 hybrid_clock.cc:648] HybridClock initialized: now 1790577079083370 us; error 35 us; skew 500 ppm
I20260928 06:31:19.084734 29459 webserver.cc:460] Webserver started at http://127.24.11.130:43815/ using document root <none> and password file <none>
I20260928 06:31:19.084995 29459 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:19.085065 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:19.085218 29459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:19.086370 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "97d56a9109fa4294ac8e549f4e68f25a"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "725c9e68a24aefef1388b36b54ef77da"
server_key_iv: "fab8f84121018c291ae721ec47039f58"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.086740 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "97d56a9109fa4294ac8e549f4e68f25a"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "725c9e68a24aefef1388b36b54ef77da"
server_key_iv: "fab8f84121018c291ae721ec47039f58"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.088132 29459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:19.089037 29473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.089224 29459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:19.089316 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "97d56a9109fa4294ac8e549f4e68f25a"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "725c9e68a24aefef1388b36b54ef77da"
server_key_iv: "fab8f84121018c291ae721ec47039f58"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.089421 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:19.092799 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:19.093071 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:19.093222 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:19.093439 29459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:19.093771 29459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:19.093827 29459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.093859 29459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:19.093888 29459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.099690 29459 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:45985
I20260928 06:31:19.099730 29586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:45985 every 8 connection(s)
I20260928 06:31:19.100076 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:19.103434 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.100854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:54393 (local address 127.24.11.190:34007)
0928 06:31:19.100936 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.100939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.101136 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:31:19.101684 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.101685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.101691 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.101737 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.103165 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.103167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.103167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.103171 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.103187 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.103192 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.103242 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.103350 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.103381 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:31:19.103672 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.100816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:34007 (local address 127.24.11.130:54393)
0928 06:31:19.101022 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.101055 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.101584 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.101815 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.101819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.101831 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.102980 (+ 1149us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.102980 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.103024 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.103063 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.103262 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.103268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.103270 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.103356 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.103424 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260928 06:31:19.104206 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29459
I20260928 06:31:19.104251 29587 heartbeater.cc:347] Connected to a master server at 127.24.11.190:34007
I20260928 06:31:19.104331 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:19.104365 29587 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:19.104529 24622 external_mini_cluster.cc:1428] Setting key 5876b4428860c5c539a299417ec55df0
I20260928 06:31:19.104534 29587 heartbeater.cc:511] Master 127.24.11.190:34007 requested a full tablet report, sending...
I20260928 06:31:19.104905 29268 ts_manager.cc:194] Registered new tserver with Master: 97d56a9109fa4294ac8e549f4e68f25a (127.24.11.130:45985)
I20260928 06:31:19.105734 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--builtin_ntp_servers=127.24.11.148:46819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:31:19.208793 29590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:19.209007 29590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:19.209028 29590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:19.210160 29590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:19.210197 29590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:19.210223 29590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:19.211480 29590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:34007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:19.211663 29590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:19.211898 29590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:19.216799 29598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.216778 29595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.216917 29596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.217149 29590 server_base.cc:1034] running on GCE node
I20260928 06:31:19.217361 29590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:19.217664 29590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:19.218837 29590 hybrid_clock.cc:648] HybridClock initialized: now 1790577079218824 us; error 30 us; skew 500 ppm
I20260928 06:31:19.220088 29590 webserver.cc:460] Webserver started at http://127.24.11.131:37403/ using document root <none> and password file <none>
I20260928 06:31:19.220346 29590 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:19.220425 29590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:19.220584 29590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:19.221732 29590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "6771371a25de404ea5efb160e9db8a65"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "425eb579b7a8d3092d1e8e7baa284adb"
server_key_iv: "b62b6fd2b8029bef5564d6a6b6ff5881"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.222105 29590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "6771371a25de404ea5efb160e9db8a65"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "425eb579b7a8d3092d1e8e7baa284adb"
server_key_iv: "b62b6fd2b8029bef5564d6a6b6ff5881"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.223408 29590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:19.224201 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.224387 29590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:19.224485 29590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "6771371a25de404ea5efb160e9db8a65"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "425eb579b7a8d3092d1e8e7baa284adb"
server_key_iv: "b62b6fd2b8029bef5564d6a6b6ff5881"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.224567 29590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:19.227939 29590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:19.228204 29590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:19.228346 29590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:19.228564 29590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:19.228865 29590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:19.228919 29590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.228951 29590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:19.228977 29590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.235487 29590 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:46769
I20260928 06:31:19.235571 29717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:46769 every 8 connection(s)
I20260928 06:31:19.235939 29590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:19.239089 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.236727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:35835 (local address 127.24.11.190:34007)
0928 06:31:19.236791 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.236793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.237078 (+ 285us) server_negotiation.cc:407] Connection header received
0928 06:31:19.237515 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.237516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.237521 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.237650 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.238832 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.238834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.238834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.238838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.238853 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.238858 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.238926 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.239003 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.239020 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:19.239248 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.236706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:34007 (local address 127.24.11.131:35835)
0928 06:31:19.236930 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.236971 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.237423 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.237721 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.237725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.237733 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.238648 (+ 915us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.238649 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.238677 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.238705 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.238932 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.238935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.238938 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.238991 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.239037 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":111,"threads_started":1}
I20260928 06:31:19.239730 29718 heartbeater.cc:347] Connected to a master server at 127.24.11.190:34007
I20260928 06:31:19.239857 29718 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:19.240051 29718 heartbeater.cc:511] Master 127.24.11.190:34007 requested a full tablet report, sending...
I20260928 06:31:19.240461 29267 ts_manager.cc:194] Registered new tserver with Master: 6771371a25de404ea5efb160e9db8a65 (127.24.11.131:46769)
I20260928 06:31:19.241144 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29590
I20260928 06:31:19.241268 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:19.241466 24622 external_mini_cluster.cc:1428] Setting key 68749f539d82f9230734a451800260f1
I20260928 06:31:19.242292 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:19.244606 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.243632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.24.11.190:34007)
0928 06:31:19.243698 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.243701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.243811 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:31:19.243863 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.243864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.243869 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.243926 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.244433 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.244434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.244434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.244437 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.244452 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.244456 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.244506 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.244540 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.244553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:31:19.246364 29267 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:19.247900 29267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:19.251981 29727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.250186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45429 (local address 127.0.0.1:41140)
0928 06:31:19.250453 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.250462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.250495 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.250916 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.250917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.250918 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.251584 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.251585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.251591 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.251615 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.251827 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.251833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.251837 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.251897 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.251911 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":62,"threads_started":1}
I20260928 06:31:19.252017 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.250388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.24.11.129:45429)
0928 06:31:19.250574 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.250578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.250587 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:19.250611 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.250614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.250623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.250683 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.251691 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.251700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.251701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.251705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.251749 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.251774 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.251833 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.251920 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.251936 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260928 06:31:19.252086 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.250056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.24.11.130:45985)
0928 06:31:19.250221 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.250227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.250343 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:31:19.250429 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.250432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.250443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.250520 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.251744 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.251751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.251751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.251756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.251789 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.251816 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.251879 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.251990 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.252006 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260928 06:31:19.252131 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.250073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.24.11.131:46769)
0928 06:31:19.250292 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.250296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.250426 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:31:19.250456 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.250459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.250470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.250538 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.251712 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.251719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.251719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.251724 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.251803 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.251827 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.251879 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.252034 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.252048 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260928 06:31:19.252243 29726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.249976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:45985 (local address 127.0.0.1:54068)
0928 06:31:19.250241 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.250286 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.250356 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.250899 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.250906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.250913 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.251647 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.251647 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.251656 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.251674 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.251907 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.251908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.251909 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.252167 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.252181 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":71,"threads_started":1}
I20260928 06:31:19.252488 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.249992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:46769 (local address 127.0.0.1:38680)
0928 06:31:19.250294 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.250306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.250357 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.250901 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.250909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.250914 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.251571 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.251572 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.251590 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.251615 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.251907 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.251908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.251909 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.251958 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.251971 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":99,"threads_started":1}
I20260928 06:31:19.253302 29390 tablet_service.cc:1467] Processing CreateTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 (DEFAULT_TABLE table=test-table [id=1777180001ba48da8a2c79edd0568789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:19.253295 29652 tablet_service.cc:1467] Processing CreateTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 (DEFAULT_TABLE table=test-table [id=1777180001ba48da8a2c79edd0568789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:19.253659 29390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.253659 29652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.256239 29733 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Bootstrap starting.
I20260928 06:31:19.256959 29733 tablet_bootstrap.cc:654] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.256934 29521 tablet_service.cc:1467] Processing CreateTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 (DEFAULT_TABLE table=test-table [id=1777180001ba48da8a2c79edd0568789]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:19.257270 29733 log.cc:824] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.257334 29521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.257640 29732 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: Bootstrap starting.
I20260928 06:31:19.258050 29733 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: No bootstrap required, opened a new log
I20260928 06:31:19.258119 29733 ts_tablet_manager.cc:1397] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:19.258531 29732 tablet_bootstrap.cc:654] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.258839 29732 log.cc:824] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.259693 29732 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: No bootstrap required, opened a new log
I20260928 06:31:19.259763 29732 ts_tablet_manager.cc:1397] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:19.260556 29736 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Bootstrap starting.
I20260928 06:31:19.260788 29733 raft_consensus.cc:348] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.260970 29733 raft_consensus.cc:374] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.261005 29733 raft_consensus.cc:729] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6771371a25de404ea5efb160e9db8a65, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.261155 29733 consensus_queue.cc:260] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.261312 29736 tablet_bootstrap.cc:654] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.261408 29733 ts_tablet_manager.cc:1428] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:31:19.261521 29718 heartbeater.cc:503] Master 127.24.11.190:34007 was elected leader, sending a full tablet report...
I20260928 06:31:19.261828 29736 log.cc:824] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.262624 29732 raft_consensus.cc:348] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.262694 29736 tablet_bootstrap.cc:492] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: No bootstrap required, opened a new log
I20260928 06:31:19.262766 29732 raft_consensus.cc:374] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.262794 29736 ts_tablet_manager.cc:1397] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:19.262830 29732 raft_consensus.cc:729] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883ca521f2aa4ed4851446b9e01636c5, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.262940 29732 consensus_queue.cc:260] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.263203 29732 ts_tablet_manager.cc:1428] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:31:19.263248 29456 heartbeater.cc:503] Master 127.24.11.190:34007 was elected leader, sending a full tablet report...
I20260928 06:31:19.265342 29736 raft_consensus.cc:348] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.265481 29736 raft_consensus.cc:374] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.265506 29736 raft_consensus.cc:729] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97d56a9109fa4294ac8e549f4e68f25a, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.265678 29736 consensus_queue.cc:260] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.265985 29736 ts_tablet_manager.cc:1428] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:31:19.266026 29587 heartbeater.cc:503] Master 127.24.11.190:34007 was elected leader, sending a full tablet report...
I20260928 06:31:19.336354 29740 raft_consensus.cc:482] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:19.336490 29740 raft_consensus.cc:504] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.336858 29740 leader_election.cc:288] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6771371a25de404ea5efb160e9db8a65 (127.24.11.131:46769), 883ca521f2aa4ed4851446b9e01636c5 (127.24.11.129:45429)
I20260928 06:31:19.338332 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.337016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:46769 (local address 127.24.11.130:45987)
0928 06:31:19.337076 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.337089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.337133 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.337315 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.337316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.337319 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.337973 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.337973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.337980 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.337992 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.338218 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.338219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.338220 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.338273 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.338285 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:31:19.338384 29729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.337101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:45987 (local address 127.24.11.131:46769)
0928 06:31:19.337153 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.337155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.337168 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:19.337214 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.337214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.337221 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.337292 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.338108 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.338110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.338110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.338115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.338136 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.338141 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.338203 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.338270 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.338287 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:19.338555 29731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.337133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:51789 (local address 127.24.11.129:45429)
0928 06:31:19.337195 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.337197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.337337 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:31:19.337370 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.337371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.337376 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.337436 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.338277 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.338279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.338279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.338284 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.338299 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.338304 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.338365 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.338466 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.338483 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:31:19.338716 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.337024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45429 (local address 127.24.11.130:51789)
0928 06:31:19.337246 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.337254 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.337315 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.337463 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.337464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.337466 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:19.338152 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:19.338153 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:19.338157 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:19.338165 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:19.338371 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:19.338372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:19.338372 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 06:31:19.338636 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 06:31:19.338649 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":51,"thread_start_us":95,"threads_started":1}
I20260928 06:31:19.338712 29672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c7e17d67c4cbc9ed01dc1f66c75a4" candidate_uuid: "97d56a9109fa4294ac8e549f4e68f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6771371a25de404ea5efb160e9db8a65" is_pre_election: true
I20260928 06:31:19.338886 29672 raft_consensus.cc:2454] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d56a9109fa4294ac8e549f4e68f25a in term 0.
I20260928 06:31:19.339064 29410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c7e17d67c4cbc9ed01dc1f66c75a4" candidate_uuid: "97d56a9109fa4294ac8e549f4e68f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883ca521f2aa4ed4851446b9e01636c5" is_pre_election: true
I20260928 06:31:19.339148 29474 leader_election.cc:302] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6771371a25de404ea5efb160e9db8a65, 97d56a9109fa4294ac8e549f4e68f25a; no voters:
I20260928 06:31:19.339198 29410 raft_consensus.cc:2454] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97d56a9109fa4294ac8e549f4e68f25a in term 0.
I20260928 06:31:19.339298 29740 raft_consensus.cc:2781] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:19.339388 29740 raft_consensus.cc:482] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:19.339417 29740 raft_consensus.cc:3037] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.340152 29740 raft_consensus.cc:504] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.340309 29740 leader_election.cc:288] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [CANDIDATE]: Term 1 election: Requested vote from peers 6771371a25de404ea5efb160e9db8a65 (127.24.11.131:46769), 883ca521f2aa4ed4851446b9e01636c5 (127.24.11.129:45429)
I20260928 06:31:19.340452 29410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c7e17d67c4cbc9ed01dc1f66c75a4" candidate_uuid: "97d56a9109fa4294ac8e549f4e68f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "883ca521f2aa4ed4851446b9e01636c5"
I20260928 06:31:19.340452 29672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "788c7e17d67c4cbc9ed01dc1f66c75a4" candidate_uuid: "97d56a9109fa4294ac8e549f4e68f25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6771371a25de404ea5efb160e9db8a65"
I20260928 06:31:19.340531 29410 raft_consensus.cc:3037] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.340551 29672 raft_consensus.cc:3037] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.341226 29410 raft_consensus.cc:2454] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d56a9109fa4294ac8e549f4e68f25a in term 1.
I20260928 06:31:19.341370 29672 raft_consensus.cc:2454] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97d56a9109fa4294ac8e549f4e68f25a in term 1.
I20260928 06:31:19.341408 29475 leader_election.cc:302] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 883ca521f2aa4ed4851446b9e01636c5, 97d56a9109fa4294ac8e549f4e68f25a; no voters:
I20260928 06:31:19.341498 29740 raft_consensus.cc:2781] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:19.341699 29740 raft_consensus.cc:686] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 1 LEADER]: Becoming Leader. State: Replica: 97d56a9109fa4294ac8e549f4e68f25a, State: Running, Role: LEADER
I20260928 06:31:19.341841 29740 consensus_queue.cc:237] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } }
I20260928 06:31:19.342717 29267 catalog_manager.cc:5495] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 97d56a9109fa4294ac8e549f4e68f25a (127.24.11.130). New cstate: current_term: 1 leader_uuid: "97d56a9109fa4294ac8e549f4e68f25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97d56a9109fa4294ac8e549f4e68f25a" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 } health_report { overall_health: UNKNOWN } } }
W20260928 06:31:19.350600 29588 tablet.cc:2369] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:19.368017 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.366974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.24.11.130:45985)
0928 06:31:19.367042 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.367045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.367053 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:19.367091 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.367092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.367111 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.367173 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.367785 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.367787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.367787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.367792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.367808 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.367812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.367868 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.367923 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.367938 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:19.369719 29672 raft_consensus.cc:1261] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 1 FOLLOWER]: Refusing update from remote peer 97d56a9109fa4294ac8e549f4e68f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:19.369717 29410 raft_consensus.cc:1261] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 883ca521f2aa4ed4851446b9e01636c5 [term 1 FOLLOWER]: Refusing update from remote peer 97d56a9109fa4294ac8e549f4e68f25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:19.370118 29740 consensus_queue.cc:1035] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6771371a25de404ea5efb160e9db8a65" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 46769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:19.370257 29742 consensus_queue.cc:1035] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "883ca521f2aa4ed4851446b9e01636c5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:19.372551 29745 mvcc.cc:205] Tried to move back new op lower bound from 7334203717095813120 to 7334203716984934400. Current Snapshot: MvccSnapshot[applied={T|T < 7334203717095813120}]
I20260928 06:31:19.373826 29747 mvcc.cc:205] Tried to move back new op lower bound from 7334203717095813120 to 7334203716984934400. Current Snapshot: MvccSnapshot[applied={T|T < 7334203717095813120 or (T in {7334203717095813120})}]
I20260928 06:31:19.374006 29746 mvcc.cc:205] Tried to move back new op lower bound from 7334203717095813120 to 7334203716984934400. Current Snapshot: MvccSnapshot[applied={T|T < 7334203717095813120 or (T in {7334203717095813120})}]
I20260928 06:31:19.374936 29267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:19.375030 29267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:19.375743 29267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2daeb31e32747cf8bfd2f451c7d621a: Sending DeleteTablet for 3 replicas of tablet 788c7e17d67c4cbc9ed01dc1f66c75a4
I20260928 06:31:19.376107 29652 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:19 UTC) from {username='slave'} at 127.0.0.1:38680
I20260928 06:31:19.376147 29390 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:19 UTC) from {username='slave'} at 127.0.0.1:41140
I20260928 06:31:19.376185 29521 tablet_service.cc:1514] Processing DeleteTablet for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:19 UTC) from {username='slave'} at 127.0.0.1:54068
I20260928 06:31:19.376480 29754 tablet_replica.cc:330] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: stopping tablet replica
I20260928 06:31:19.376515 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29328
I20260928 06:31:19.377183 29754 raft_consensus.cc:2229] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:19.377311 29754 raft_consensus.cc:2258] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:19.381690 29756 tablet_replica.cc:330] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a: stopping tablet replica
I20260928 06:31:19.381817 29756 raft_consensus.cc:2229] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:19.381959 29756 raft_consensus.cc:2258] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 97d56a9109fa4294ac8e549f4e68f25a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:19.382179 29754 ts_tablet_manager.cc:1905] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:31:19.382603 29253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:45429 (error 108)
I20260928 06:31:19.382932 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29459
I20260928 06:31:19.383095 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.382938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45429 (local address 127.0.0.1:41142)
0928 06:31:19.383007 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.383059 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:31:19.384085 29754 ts_tablet_manager.cc:1918] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:19.384169 29754 log.cc:1196] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/788c7e17d67c4cbc9ed01dc1f66c75a4
I20260928 06:31:19.384534 29754 ts_tablet_manager.cc:1939] T 788c7e17d67c4cbc9ed01dc1f66c75a4 P 6771371a25de404ea5efb160e9db8a65: Deleting consensus metadata
W20260928 06:31:19.384668 29253 catalog_manager.cc:4584] TS 883ca521f2aa4ed4851446b9e01636c5 (127.24.11.129:45429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 788c7e17d67c4cbc9ed01dc1f66c75a4: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45429: connect: Connection refused (error 111)
I20260928 06:31:19.384994 29252 catalog_manager.cc:4841] TS 6771371a25de404ea5efb160e9db8a65 (127.24.11.131:46769): tablet 788c7e17d67c4cbc9ed01dc1f66c75a4 (table test-table [id=1777180001ba48da8a2c79edd0568789]) successfully deleted
I20260928 06:31:19.388350 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29590
I20260928 06:31:19.388435 29730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.388307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:45985 (local address 127.0.0.1:54098)
0928 06:31:19.388362 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.388390 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:45985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:31:19.394776 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29237
2026-09-28T06:31:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (691 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:31:19.414263 24622 mini_oidc.cc:152] Starting JWKS server
I20260928 06:31:19.415319 24622 webserver.cc:460] Webserver started at https://127.0.0.1:43405/ using document root <none> and password file <none>
I20260928 06:31:19.415437 24622 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:31:19.415612 24622 webserver.cc:460] Webserver started at http://127.0.0.1:41851/ using document root <none> and password file <none>
2026-09-28T06:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:19Z Disabled control of system clock
I20260928 06:31:19.423141 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34785
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:33903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:34785
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43405/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/testchainca.cert with env {}
W20260928 06:31:19.502274 29763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:19.502465 29763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:19.502487 29763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:19.503643 29763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:19.503684 29763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:19.503697 29763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:19.503710 29763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:19.503762 29763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:31:19.503787 29763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:31:19.503805 29763 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43405/jwks.json
I20260928 06:31:19.505340 29763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:34785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43405/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:19.505605 29763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:19.505851 29763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:19.510560 29769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.510564 29768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.510648 29771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.510850 29763 server_base.cc:1034] running on GCE node
I20260928 06:31:19.511098 29763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:19.511382 29763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:19.512552 29763 hybrid_clock.cc:648] HybridClock initialized: now 1790577079512541 us; error 49 us; skew 500 ppm
I20260928 06:31:19.513800 29763 webserver.cc:460] Webserver started at http://127.24.11.190:44171/ using document root <none> and password file <none>
I20260928 06:31:19.514043 29763 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:19.514111 29763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:19.514257 29763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:19.515352 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "ee527d56c28f4c6ca8994b5b46a36683"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "09563a508a691b1ffdbfb2bbad8927c6"
server_key_iv: "264bc1dd0549f28822b0210f981245d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.515718 29763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "ee527d56c28f4c6ca8994b5b46a36683"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "09563a508a691b1ffdbfb2bbad8927c6"
server_key_iv: "264bc1dd0549f28822b0210f981245d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.516978 29763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:19.517715 29777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.517894 29763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:19.517989 29763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "ee527d56c28f4c6ca8994b5b46a36683"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "09563a508a691b1ffdbfb2bbad8927c6"
server_key_iv: "264bc1dd0549f28822b0210f981245d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.518074 29763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:19.543166 29763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:19.543524 29763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:19.543689 29763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:19.548084 29763 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:34785
I20260928 06:31:19.548118 29829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:34785 every 8 connection(s)
I20260928 06:31:19.548482 29763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:19.549419 29830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.551652 29830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683: Bootstrap starting.
I20260928 06:31:19.552182 29830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.552415 29830 log.cc:824] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.553045 29830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683: No bootstrap required, opened a new log
I20260928 06:31:19.554611 29830 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } }
I20260928 06:31:19.554764 29830 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.554788 29830 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee527d56c28f4c6ca8994b5b46a36683, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.554932 29830 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } }
I20260928 06:31:19.555015 29830 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:19.555037 29830 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:19.555065 29830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.555514 29830 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } }
I20260928 06:31:19.555598 29830 leader_election.cc:302] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee527d56c28f4c6ca8994b5b46a36683; no voters:
I20260928 06:31:19.555712 29830 leader_election.cc:288] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:19.555809 29833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:19.555965 29833 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [term 1 LEADER]: Becoming Leader. State: Replica: ee527d56c28f4c6ca8994b5b46a36683, State: Running, Role: LEADER
I20260928 06:31:19.556018 29830 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:19.556082 29833 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } }
I20260928 06:31:19.556494 29834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee527d56c28f4c6ca8994b5b46a36683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } } }
I20260928 06:31:19.556593 29834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:19.556581 29835 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee527d56c28f4c6ca8994b5b46a36683. Latest consensus state: current_term: 1 leader_uuid: "ee527d56c28f4c6ca8994b5b46a36683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee527d56c28f4c6ca8994b5b46a36683" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34785 } } }
I20260928 06:31:19.556667 29835 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:19.557109 29842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:19.557664 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29763
I20260928 06:31:19.557682 29842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:19.557775 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:19.557973 24622 external_mini_cluster.cc:1428] Setting key 237c107aa0433135d795989187a30dec
I20260928 06:31:19.559280 29842 catalog_manager.cc:1347] Generated new cluster ID: 8c3fca343fc64e16a1cadedaae624c17
I20260928 06:31:19.559335 29842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:19.564054 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.558400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.24.11.190:34785)
0928 06:31:19.558576 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.558587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.558743 (+ 156us) server_negotiation.cc:407] Connection header received
0928 06:31:19.558874 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.558902 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.558986 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.559124 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.559399 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.560743 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.561885 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.562504 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.562611 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.563371 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.563374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.563375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.563575 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.563614 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.563628 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.563690 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.563796 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.563865 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
I20260928 06:31:19.564962 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:19.572961 29842 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:19.573467 29842 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:19.577594 29842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee527d56c28f4c6ca8994b5b46a36683: Generated new TSK 0
I20260928 06:31:19.577773 29842 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:31:19.609333 29860 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:31:19.609505 29860 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:31:19.609592 29852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.586838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.24.11.190:34785)
0928 06:31:19.586893 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.586896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.587027 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:31:19.587158 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.587158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.587171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.587231 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:31:19.587381 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.587867 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.588647 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.588855 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.588946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.589157 (+ 211us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:31:19.609306 (+ 20149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:31:19.609483 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:31:19.609737 29852 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 06:31:19.610226 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:19.586776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:34785 (local address 127.0.0.1:52666)
0928 06:31:19.586959 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.586968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.587049 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.587237 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.587238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.587240 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:31:19.587307 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.587312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.587967 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.587968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.588560 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.588562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.588963 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.588964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.589031 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.589048 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:31:19.609558 (+ 20510us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:31:19.609981 (+ 423us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.11.190:34785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.11.190:34785: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43405/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:31:19.610673 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29763
2026-09-28T06:31:19Z chronyd exiting
I20260928 06:31:19.647009 24622 test_util.cc:175] -----------------------------------------------
I20260928 06:31:19.647094 24622 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577055356400-24622-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:31:19.647863 24622 mini_oidc.cc:152] Starting JWKS server
I20260928 06:31:19.648571 24622 webserver.cc:460] Webserver started at https://127.0.0.1:40135/ using document root <none> and password file <none>
I20260928 06:31:19.648613 24622 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:31:19.648808 24622 webserver.cc:460] Webserver started at http://127.0.0.1:43653/ using document root <none> and password file <none>
2026-09-28T06:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:19Z Disabled control of system clock
I20260928 06:31:19.656162 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43991
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:43433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40135/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/testchainca.cert with env {}
W20260928 06:31:19.737115 29866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:19.737372 29866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:19.737412 29866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:19.738621 29866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:19.738691 29866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:19.738713 29866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:19.738734 29866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:19.738771 29866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:31:19.738798 29866 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:31:19.738826 29866 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40135/jwks.json
I20260928 06:31:19.740423 29866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:43433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40135/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:19.740703 29866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:19.740934 29866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:19.745776 29872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.745805 29874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.745831 29871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.745970 29866 server_base.cc:1034] running on GCE node
I20260928 06:31:19.746209 29866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:19.746520 29866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:19.747684 29866 hybrid_clock.cc:648] HybridClock initialized: now 1790577079747667 us; error 58 us; skew 500 ppm
I20260928 06:31:19.749176 29866 webserver.cc:460] Webserver started at http://127.24.11.190:41769/ using document root <none> and password file <none>
I20260928 06:31:19.749415 29866 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:19.749486 29866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:19.749658 29866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:19.750751 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "da68b875930b4f1983f9a7510d08c458"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "7e718897329f19bdff753b6c7e836744"
server_key_iv: "34cdec4179cd8953ba686370c7b91d1b"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.751142 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "da68b875930b4f1983f9a7510d08c458"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "7e718897329f19bdff753b6c7e836744"
server_key_iv: "34cdec4179cd8953ba686370c7b91d1b"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.752596 29866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:19.753597 29880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.753796 29866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:19.753933 29866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "da68b875930b4f1983f9a7510d08c458"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "7e718897329f19bdff753b6c7e836744"
server_key_iv: "34cdec4179cd8953ba686370c7b91d1b"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.754021 29866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:19.763072 29866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:19.763417 29866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:19.763586 29866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:19.767376 29866 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43991
I20260928 06:31:19.767427 29932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43991 every 8 connection(s)
I20260928 06:31:19.767756 29866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:19.768425 29933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.770298 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29866
I20260928 06:31:19.770381 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:19.770562 24622 external_mini_cluster.cc:1428] Setting key 545ba2bd18b53397d55f114654a94d6e
I20260928 06:31:19.771442 29933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: Bootstrap starting.
I20260928 06:31:19.772017 29933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.772357 29933 log.cc:824] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.773041 29933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: No bootstrap required, opened a new log
I20260928 06:31:19.774863 29933 raft_consensus.cc:348] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } }
I20260928 06:31:19.775045 29933 raft_consensus.cc:374] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.775089 29933 raft_consensus.cc:729] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da68b875930b4f1983f9a7510d08c458, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.775255 29933 consensus_queue.cc:260] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } }
I20260928 06:31:19.775326 29933 raft_consensus.cc:388] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:19.775348 29933 raft_consensus.cc:482] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:19.775441 29933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.776104 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.771119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.24.11.190:43991)
0928 06:31:19.771318 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.771326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.771362 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:31:19.771809 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.771816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.771889 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.772042 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.772228 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.773193 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.774049 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.774466 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.774605 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.775416 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:19.775420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:19.775421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:19.775643 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:19.775677 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:19.775689 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:19.775759 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:19.775837 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:31:19.775945 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260928 06:31:19.776093 29933 raft_consensus.cc:504] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } }
I20260928 06:31:19.776222 29933 leader_election.cc:302] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da68b875930b4f1983f9a7510d08c458; no voters:
I20260928 06:31:19.776399 29933 leader_election.cc:288] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:19.776436 29938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:19.776608 29938 raft_consensus.cc:686] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [term 1 LEADER]: Becoming Leader. State: Replica: da68b875930b4f1983f9a7510d08c458, State: Running, Role: LEADER
I20260928 06:31:19.776647 29933 sys_catalog.cc:564] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:19.776727 29938 consensus_queue.cc:237] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } }
I20260928 06:31:19.777033 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:19.777127 29938 sys_catalog.cc:455] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da68b875930b4f1983f9a7510d08c458. Latest consensus state: current_term: 1 leader_uuid: "da68b875930b4f1983f9a7510d08c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } } }
I20260928 06:31:19.777197 29938 sys_catalog.cc:458] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:19.777139 29939 sys_catalog.cc:455] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da68b875930b4f1983f9a7510d08c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da68b875930b4f1983f9a7510d08c458" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43991 } } }
I20260928 06:31:19.777241 29939 sys_catalog.cc:458] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458 [sys.catalog]: This master's current role is: LEADER
W20260928 06:31:19.778496 29954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:19.778539 29954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:19.778605 29944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:19.778784 29944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:19.780326 29944 catalog_manager.cc:1347] Generated new cluster ID: c95c0adf64bf411792cb123d7571020d
I20260928 06:31:19.780390 29944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:19.798946 29944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:19.799515 29944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:19.808881 29944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da68b875930b4f1983f9a7510d08c458: Generated new TSK 0
I20260928 06:31:19.809134 29944 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:31:19.834403 29963 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:31:19.834498 29963 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:31:19.834831 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.820366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.24.11.190:43991)
0928 06:31:19.820413 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.820415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.820515 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:31:19.820573 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.820574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.820586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.820659 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:31:19.820773 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.821590 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.822384 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.822693 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.822764 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.822941 (+ 177us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:31:19.834589 (+ 11648us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:31:19.834759 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:31:19.834901 29936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:31:19.834937 29962 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:19.820266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43991 (local address 127.0.0.1:57402)
0928 06:31:19.820449 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:19.820457 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:19.820507 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:19.820661 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:19.820662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:19.820663 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:31:19.820708 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.820710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.821694 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.821695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.822307 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:19.822309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.822770 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.822771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:19.822811 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:19.822815 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:31:19.834706 (+ 11891us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:31:19.834810 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.11.190:43991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40135/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":91,"threads_started":1}
I20260928 06:31:19.835292 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29866
2026-09-28T06:31:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:31:19.872794 24622 mini_oidc.cc:152] Starting JWKS server
I20260928 06:31:19.873586 24622 webserver.cc:460] Webserver started at https://127.0.0.1:39391/ using document root <none> and password file <none>
I20260928 06:31:19.873658 24622 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:31:19.873821 24622 webserver.cc:460] Webserver started at http://127.0.0.1:44101/ using document root <none> and password file <none>
2026-09-28T06:31:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:19Z Disabled control of system clock
I20260928 06:31:19.881582 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35315
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:41773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39391/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:31:19.960952 29969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:19.961153 29969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:19.961176 29969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:19.962292 29969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:19.962339 29969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:19.962352 29969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:19.962365 29969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:19.962378 29969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:31:19.962389 29969 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:31:19.962404 29969 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39391/jwks.json
I20260928 06:31:19.963685 29969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:41773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39391/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:19.963876 29969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:19.964098 29969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:19.968883 29975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.968945 29974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:19.968883 29977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:19.969278 29969 server_base.cc:1034] running on GCE node
I20260928 06:31:19.969558 29969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:19.969861 29969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:19.971004 29969 hybrid_clock.cc:648] HybridClock initialized: now 1790577079970987 us; error 33 us; skew 500 ppm
I20260928 06:31:19.972242 29969 webserver.cc:460] Webserver started at http://127.24.11.190:44361/ using document root <none> and password file <none>
I20260928 06:31:19.972488 29969 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:19.972535 29969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:19.972682 29969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:19.973855 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "1b69d4e839394e0899a5851c36bc6d8e"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "b524d9d7a65fecb6a315f2f9250b2112"
server_key_iv: "f12979b24946087907e4c63a5b98aa8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.974359 29969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "1b69d4e839394e0899a5851c36bc6d8e"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "b524d9d7a65fecb6a315f2f9250b2112"
server_key_iv: "f12979b24946087907e4c63a5b98aa8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.975735 29969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:19.976491 29983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:19.976678 29969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:19.976800 29969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "1b69d4e839394e0899a5851c36bc6d8e"
format_stamp: "Formatted at 2026-09-28 06:31:19 on dist-test-slave-38pg"
server_key: "b524d9d7a65fecb6a315f2f9250b2112"
server_key_iv: "f12979b24946087907e4c63a5b98aa8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:19.976881 29969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:19.984761 29969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:19.985055 29969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:19.985203 29969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:19.988991 29969 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:35315
I20260928 06:31:19.989022 30035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:35315 every 8 connection(s)
I20260928 06:31:19.989353 29969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:19.990079 30036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:19.992871 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e: Bootstrap starting.
I20260928 06:31:19.993455 30036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:19.993758 30036 log.cc:824] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:19.994513 30036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e: No bootstrap required, opened a new log
I20260928 06:31:19.995821 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 29969
I20260928 06:31:19.995911 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:19.996095 24622 external_mini_cluster.cc:1428] Setting key 9f0ef3fd8c75c69c893fd8d30f210b38
I20260928 06:31:19.996294 30036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } }
I20260928 06:31:19.996452 30036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:19.996516 30036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b69d4e839394e0899a5851c36bc6d8e, State: Initialized, Role: FOLLOWER
I20260928 06:31:19.996665 30036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } }
I20260928 06:31:19.996766 30036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:19.996829 30036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:19.996886 30036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:19.997639 30036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } }
I20260928 06:31:19.997748 30036 leader_election.cc:302] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b69d4e839394e0899a5851c36bc6d8e; no voters:
I20260928 06:31:19.997905 30036 leader_election.cc:288] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:19.998148 30036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:19.998175 30041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:19.998296 30041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [term 1 LEADER]: Becoming Leader. State: Replica: 1b69d4e839394e0899a5851c36bc6d8e, State: Running, Role: LEADER
I20260928 06:31:19.998402 30041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } }
I20260928 06:31:19.998729 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b69d4e839394e0899a5851c36bc6d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } } }
I20260928 06:31:19.998809 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:19.998947 30042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b69d4e839394e0899a5851c36bc6d8e. Latest consensus state: current_term: 1 leader_uuid: "1b69d4e839394e0899a5851c36bc6d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b69d4e839394e0899a5851c36bc6d8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35315 } } }
I20260928 06:31:19.999013 30042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:19.999136 30050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:19.999785 30050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:20.001662 30050 catalog_manager.cc:1347] Generated new cluster ID: c2ba4c1112ed42d292904c835a761d69
I20260928 06:31:20.001714 30050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:20.002327 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:19.996583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.24.11.190:35315)
0928 06:31:19.996827 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:19.996836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:19.996873 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:31:19.997002 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:19.997010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:19.997079 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:19.997231 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:19.997417 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:19.998870 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:19.999990 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.000563 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.000689 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.001429 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:20.001433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:20.001433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:20.001687 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:20.001734 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:20.001749 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:20.001812 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:20.002059 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:31:20.002133 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260928 06:31:20.003290 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:20.013850 30050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:20.014568 30050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:20.029201 30050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b69d4e839394e0899a5851c36bc6d8e: Generated new TSK 0
I20260928 06:31:20.029431 30050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:20.047055 30040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.043208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.24.11.190:35315)
0928 06:31:20.043278 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:20.043280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:20.043397 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:31:20.043476 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:20.043477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:20.043488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:20.043553 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:31:20.043711 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.044213 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.045355 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.045617 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.045672 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.045879 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:31:20.046839 (+ 960us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39391/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:31:20.046982 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39391/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 06:31:20.047127 30040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39391/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:31:20.047173 30065 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:20.043148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35315 (local address 127.0.0.1:60140)
0928 06:31:20.043308 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.043320 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:20.043400 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:20.043575 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:20.043575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:20.043578 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:31:20.043645 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.043647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.044298 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.044299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.045265 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.045268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.045693 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.045695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.045766 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.045772 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:31:20.046955 (+ 1183us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39391/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:31:20.047071 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.11.190:35315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39391/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260928 06:31:20.047420 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 29969
E20260928 06:31:20.058022 30066 webserver.cc:533] Webserver: sslize failed:
E20260928 06:31:20.058102 30066 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:31:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:20Z Disabled control of system clock
I20260928 06:31:20.091943 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39047
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39047
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/testchainca.cert with env {}
W20260928 06:31:20.171298 30070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:20.171517 30070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:20.171540 30070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:20.172631 30070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:20.172667 30070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:20.172680 30070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:20.172693 30070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:20.172705 30070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:31:20.172717 30070 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:31:20.172729 30070 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:31:20.174271 30070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:20.174481 30070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:20.174649 30070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:20.179301 30076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:20.179299 30075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:20.179435 30078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:20.179504 30070 server_base.cc:1034] running on GCE node
I20260928 06:31:20.179724 30070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:20.180049 30070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:20.181216 30070 hybrid_clock.cc:648] HybridClock initialized: now 1790577080181202 us; error 46 us; skew 500 ppm
I20260928 06:31:20.182533 30070 webserver.cc:460] Webserver started at http://127.24.11.190:38185/ using document root <none> and password file <none>
I20260928 06:31:20.182773 30070 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:20.182842 30070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:20.182997 30070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:20.184124 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d"
format_stamp: "Formatted at 2026-09-28 06:31:20 on dist-test-slave-38pg"
server_key: "f705fc2d48ee5a35b8cd124f24ced050"
server_key_iv: "a3cbe2ed1f1fdbe24b8c30ff34ce4d5d"
server_key_version: "encryptionkey@0"
I20260928 06:31:20.184499 30070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d"
format_stamp: "Formatted at 2026-09-28 06:31:20 on dist-test-slave-38pg"
server_key: "f705fc2d48ee5a35b8cd124f24ced050"
server_key_iv: "a3cbe2ed1f1fdbe24b8c30ff34ce4d5d"
server_key_version: "encryptionkey@0"
I20260928 06:31:20.185966 30070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:20.186686 30084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:20.186890 30070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:20.186996 30070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d"
format_stamp: "Formatted at 2026-09-28 06:31:20 on dist-test-slave-38pg"
server_key: "f705fc2d48ee5a35b8cd124f24ced050"
server_key_iv: "a3cbe2ed1f1fdbe24b8c30ff34ce4d5d"
server_key_version: "encryptionkey@0"
I20260928 06:31:20.187069 30070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:20.240716 30070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:20.241074 30070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:20.241233 30070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:20.245195 30070 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:39047
I20260928 06:31:20.245230 30136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:39047 every 8 connection(s)
I20260928 06:31:20.245626 30070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:20.246688 30137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:20.246712 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30070
I20260928 06:31:20.246815 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:20.247004 24622 external_mini_cluster.cc:1428] Setting key dd2fd60762c4701f92e738650ee4fa7a
I20260928 06:31:20.249889 30137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d: Bootstrap starting.
I20260928 06:31:20.250576 30137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:20.250897 30137 log.cc:824] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:20.251573 30137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d: No bootstrap required, opened a new log
I20260928 06:31:20.252642 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.247532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.24.11.190:39047)
0928 06:31:20.247727 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:20.247734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:20.247759 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:20.248209 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:20.248216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:20.248278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:20.248394 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:20.248624 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.249485 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.250590 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.251077 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.251213 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.251958 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:20.251961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:20.251962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:20.252127 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:20.252160 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:20.252171 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:20.252225 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:20.252330 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:31:20.252390 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260928 06:31:20.253661 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:31:20.253805 30137 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } }
I20260928 06:31:20.253963 30137 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:20.253994 30137 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c5a61cbeb246aa8ca6aa2cc957255d, State: Initialized, Role: FOLLOWER
I20260928 06:31:20.254113 30137 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } }
I20260928 06:31:20.254194 30137 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:20.254225 30137 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:20.254271 30137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:20.254818 30137 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } }
I20260928 06:31:20.254904 30137 leader_election.cc:302] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9c5a61cbeb246aa8ca6aa2cc957255d; no voters:
I20260928 06:31:20.255029 30137 leader_election.cc:288] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:20.255100 30142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:20.255266 30142 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [term 1 LEADER]: Becoming Leader. State: Replica: e9c5a61cbeb246aa8ca6aa2cc957255d, State: Running, Role: LEADER
I20260928 06:31:20.255316 30137 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:20.255353 30142 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } }
I20260928 06:31:20.255762 30143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } } }
I20260928 06:31:20.255776 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9c5a61cbeb246aa8ca6aa2cc957255d. Latest consensus state: current_term: 1 leader_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9c5a61cbeb246aa8ca6aa2cc957255d" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39047 } } }
I20260928 06:31:20.255950 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:20.255929 30143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:20.256323 30151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:20.256816 30151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:20.258523 30151 catalog_manager.cc:1347] Generated new cluster ID: 75b3b5f334b240368644a62d85234327
I20260928 06:31:20.258592 30151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:20.270514 30151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:20.271191 30151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:20.275185 30151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9c5a61cbeb246aa8ca6aa2cc957255d: Generated new TSK 0
I20260928 06:31:20.275390 30151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:20.323220 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:20.283209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.24.11.190:39047)
0928 06:31:20.283273 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:20.283276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:20.283402 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:31:20.283484 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:20.283485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:20.283496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:20.283556 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:31:20.283696 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.284187 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.285306 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.285505 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.285625 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.285795 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:31:20.322999 (+ 37204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:31:20.323160 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40764: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 06:31:20.323345 30140 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40764: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:31:20.323423 30166 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:20.283154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39047 (local address 127.0.0.1:40764)
0928 06:31:20.283328 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:20.283340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:20.283408 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:20.283573 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:20.283574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:20.283576 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:31:20.283632 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.283634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.284263 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.284265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.285222 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:20.285225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:20.285629 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:20.285630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:20.285692 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:20.285698 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:31:20.323158 (+ 37460us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:31:20.323303 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.11.190:39047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260928 06:31:20.323698 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30070
2026-09-28T06:31:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (256 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:20 dist-test-slave-38pg krb5kdc[30178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:20 dist-test-slave-38pg krb5kdc[30178](info): set up 2 sockets
Sep 28 06:31:20 dist-test-slave-38pg krb5kdc[30178](info): commencing operation
krb5kdc: starting...
W20260928 06:31:21.578807 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:21 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:21.696826 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35395
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:21.779487 30191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:21.779719 30191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:21.779743 30191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:21.780932 30191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:21.780973 30191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:21.780988 30191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:21.781000 30191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:21.781013 30191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:21.782603 30191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:21.782814 30191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:21.782998 30191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:21.787927 30197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:21.787941 30196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:21.787931 30199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:21.788235 30191 server_base.cc:1034] running on GCE node
I20260928 06:31:21.788467 30191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:21.788754 30191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:21.789921 30191 hybrid_clock.cc:648] HybridClock initialized: now 1790577081789901 us; error 39 us; skew 500 ppm
Sep 28 06:31:21 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577081, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:21.792507 30191 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:21.792996 30191 webserver.cc:460] Webserver started at http://127.24.11.190:34541/ using document root <none> and password file <none>
I20260928 06:31:21.793243 30191 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:21.793291 30191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:21.793437 30191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:21.794549 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "815b74df5a534a69834bb3ed66353aec"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "f800749eccd9d64a099bb82a30c73ec2"
server_key_iv: "d40c9d8b4ede46806cf5a85b42488841"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.794967 30191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "815b74df5a534a69834bb3ed66353aec"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "f800749eccd9d64a099bb82a30c73ec2"
server_key_iv: "d40c9d8b4ede46806cf5a85b42488841"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.796296 30191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:21.797036 30206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:21.797230 30191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:21.797324 30191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "815b74df5a534a69834bb3ed66353aec"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "f800749eccd9d64a099bb82a30c73ec2"
server_key_iv: "d40c9d8b4ede46806cf5a85b42488841"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.797401 30191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:21.821178 30191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:21.823258 30191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:21.823415 30191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:21.827265 30191 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:35395
I20260928 06:31:21.827348 30258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:35395 every 8 connection(s)
I20260928 06:31:21.827678 30191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:21.828528 30259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:21.830801 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec: Bootstrap starting.
I20260928 06:31:21.831398 30259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:21.831660 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30191
I20260928 06:31:21.831776 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:21.831789 30259 log.cc:824] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:21.831969 24622 external_mini_cluster.cc:1428] Setting key d22a5eb4e6f3fc6023b192001aed14e8
I20260928 06:31:21.832665 30259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec: No bootstrap required, opened a new log
I20260928 06:31:21.835337 30259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } }
I20260928 06:31:21.835525 30259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:21.835561 30259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815b74df5a534a69834bb3ed66353aec, State: Initialized, Role: FOLLOWER
I20260928 06:31:21.835681 30259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } }
I20260928 06:31:21.835748 30259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:21.835775 30259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:21.835819 30259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:21.836447 30259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } }
I20260928 06:31:21.836536 30259 leader_election.cc:302] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815b74df5a534a69834bb3ed66353aec; no voters:
I20260928 06:31:21.836717 30259 leader_election.cc:288] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:21.836748 30264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:21.836908 30264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [term 1 LEADER]: Becoming Leader. State: Replica: 815b74df5a534a69834bb3ed66353aec, State: Running, Role: LEADER
I20260928 06:31:21.836987 30259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:21.836997 30264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } }
I20260928 06:31:21.837317 30264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815b74df5a534a69834bb3ed66353aec. Latest consensus state: current_term: 1 leader_uuid: "815b74df5a534a69834bb3ed66353aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } } }
I20260928 06:31:21.837419 30264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:21.837332 30265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "815b74df5a534a69834bb3ed66353aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815b74df5a534a69834bb3ed66353aec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35395 } } }
I20260928 06:31:21.837572 30265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:21.838291 30270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:21.838901 30270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:31:21 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:21.840931 30270 catalog_manager.cc:1347] Generated new cluster ID: 324b207b3845420391e581a9edc7e3c3
I20260928 06:31:21.840988 30270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:21.843327 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:21.832542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.24.11.190:35395)
0928 06:31:21.832709 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:21.832719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:21.832754 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:31:21.833185 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:21.833191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:21.833238 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:21.833355 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:21.834181 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:21.835060 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:21.836090 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:21.836496 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:21.836651 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:21.839726 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:21.839729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:21.839730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:21.839742 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:21.841697 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:21.842117 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:21.842120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:21.842120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:21.842220 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:21.842432 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:21.842433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:21.842433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:21.842714 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:21.842859 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:21.842996 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:31:21.843124 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260928 06:31:21.855326 30270 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:21.856081 30270 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:21.860677 30270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 815b74df5a534a69834bb3ed66353aec: Generated new TSK 0
I20260928 06:31:21.860891 30270 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:21.888319 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:21.971190 30285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:21.971433 30285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:21.971470 30285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:21.972645 30285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:21.972710 30285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:21.972754 30285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:21.974255 30285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:21.974514 30285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:21.974753 30285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:21.979717 30293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:21.979800 30290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:21.979837 30291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:21.979873 30285 server_base.cc:1034] running on GCE node
I20260928 06:31:21.980181 30285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:21.980423 30285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:21.981588 30285 hybrid_clock.cc:648] HybridClock initialized: now 1790577081981576 us; error 30 us; skew 500 ppm
Sep 28 06:31:21 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577081, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:21.984051 30285 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:21.984503 30285 webserver.cc:460] Webserver started at http://127.24.11.129:36359/ using document root <none> and password file <none>
I20260928 06:31:21.984722 30285 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:21.984766 30285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:21.984920 30285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:21.986027 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "14206fd348f64a18988c7e4f1ba1c7c0"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "d36c4554d7c41de0fa72edc8710b7943"
server_key_iv: "d8fdd38f3dd8f3db3dcd6640ea738429"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.986379 30285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "14206fd348f64a18988c7e4f1ba1c7c0"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "d36c4554d7c41de0fa72edc8710b7943"
server_key_iv: "d8fdd38f3dd8f3db3dcd6640ea738429"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.987658 30285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:21.988350 30300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:21.988523 30285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:21.988627 30285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "14206fd348f64a18988c7e4f1ba1c7c0"
format_stamp: "Formatted at 2026-09-28 06:31:21 on dist-test-slave-38pg"
server_key: "d36c4554d7c41de0fa72edc8710b7943"
server_key_iv: "d8fdd38f3dd8f3db3dcd6640ea738429"
server_key_version: "encryptionkey@0"
I20260928 06:31:21.988700 30285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:22.004704 30285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:22.006994 30285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:22.007158 30285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:22.007398 30285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:22.007711 30285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:22.007766 30285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.007808 30285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:22.007833 30285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.013705 30285 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:41927
I20260928 06:31:22.013754 30413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:41927 every 8 connection(s)
I20260928 06:31:22.014078 30285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577081, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:22.023499 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30285
I20260928 06:31:22.023609 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:22.023823 24622 external_mini_cluster.cc:1428] Setting key f9466f7efdee37cad058c7e25b215369
I20260928 06:31:22.023955 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.014845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33089 (local address 127.24.11.190:35395)
0928 06:31:22.014934 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.014936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.015165 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:31:22.016077 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.016078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.016089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.016150 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.016805 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.017400 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.018471 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.018669 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.018740 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.020808 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.020809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.020809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.020816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.022413 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.022851 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.022851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.022852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.022903 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.023131 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.023132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.023132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.023234 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.023471 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.023752 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.023834 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 06:31:22.024163 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.014819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35395 (local address 127.24.11.129:33089)
0928 06:31:22.015012 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.015051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:22.015950 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:22.016226 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:22.016230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:22.016489 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:22.016677 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.016681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.017483 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.017484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.018357 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.018359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.018755 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.018756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.018854 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.019364 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:22.019372 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:22.020728 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:22.022501 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.022508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.022510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.022740 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.022976 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.022977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.022977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.023077 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.023346 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:22.023352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:22.023527 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 06:31:22.023686 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 06:31:22.023919 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":112,"threads_started":1}
I20260928 06:31:22.024650 30414 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35395
I20260928 06:31:22.024745 30414 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:22.024941 30414 heartbeater.cc:511] Master 127.24.11.190:35395 requested a full tablet report, sending...
I20260928 06:31:22.025447 30223 ts_manager.cc:194] Registered new tserver with Master: 14206fd348f64a18988c7e4f1ba1c7c0 (127.24.11.129:41927)
I20260928 06:31:22.026103 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:33089
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:22.049250 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:22.129626 30419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:22.129832 30419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:22.129863 30419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:22.130966 30419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:22.130999 30419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:22.131026 30419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:22.132421 30419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:22.132607 30419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:22.132782 30419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:22.137720 30427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:22.137728 30425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:22.137847 30424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:22.138091 30419 server_base.cc:1034] running on GCE node
I20260928 06:31:22.138345 30419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:22.138585 30419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:22.139727 30419 hybrid_clock.cc:648] HybridClock initialized: now 1790577082139702 us; error 41 us; skew 500 ppm
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.142233 30419 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:22.142688 30419 webserver.cc:460] Webserver started at http://127.24.11.130:34565/ using document root <none> and password file <none>
I20260928 06:31:22.142889 30419 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:22.142931 30419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:22.143019 30419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:22.144053 30419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "f96bd003b8d447768899b4a6fb2a5c88"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "df2867aaee2292aab1e5fec6fc6a6bcd"
server_key_iv: "d8a8c5886439c5a089ff9634c2fbe474"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.144340 30419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "f96bd003b8d447768899b4a6fb2a5c88"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "df2867aaee2292aab1e5fec6fc6a6bcd"
server_key_iv: "d8a8c5886439c5a089ff9634c2fbe474"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.145632 30419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:22.146377 30434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.146553 30419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:22.146658 30419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f96bd003b8d447768899b4a6fb2a5c88"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "df2867aaee2292aab1e5fec6fc6a6bcd"
server_key_iv: "d8a8c5886439c5a089ff9634c2fbe474"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.146733 30419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:22.161010 30419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:22.163203 30419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:22.163383 30419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:22.163631 30419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:22.164001 30419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:22.164057 30419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.164098 30419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:22.164125 30419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.170120 30419 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:42221
I20260928 06:31:22.170161 30547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:42221 every 8 connection(s)
I20260928 06:31:22.170486 30419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:22.174264 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30419
I20260928 06:31:22.174410 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:22.174587 24622 external_mini_cluster.cc:1428] Setting key f5024d80c408b8809bcfd4ecd64041e7
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:22.180578 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.171245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:50085 (local address 127.24.11.190:35395)
0928 06:31:22.171315 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.171317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.171498 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:31:22.172393 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.172394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.172404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.172467 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.173062 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.173569 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.174691 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.174906 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.175010 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.177360 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.177361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.177362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.177370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.179082 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.179509 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.179510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.179511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.179574 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.179832 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.179833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.179834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.180010 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.180093 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.180344 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.180441 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:22.180586 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.171206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35395 (local address 127.24.11.130:50085)
0928 06:31:22.171362 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.171403 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:22.172289 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:22.172524 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:22.172527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:22.172783 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:22.172986 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.172990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.173673 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.173675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.174592 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.174595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.175211 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.175212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.175318 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.175882 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:22.175891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:22.177257 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:22.179180 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.179194 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.179196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.179424 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.179645 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.179646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.179646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.179753 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.180086 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:22.180090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:22.180208 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 06:31:22.180295 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:31:22.180422 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260928 06:31:22.181162 30548 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35395
I20260928 06:31:22.181282 30548 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:22.181469 30548 heartbeater.cc:511] Master 127.24.11.190:35395 requested a full tablet report, sending...
I20260928 06:31:22.181900 30223 ts_manager.cc:194] Registered new tserver with Master: f96bd003b8d447768899b4a6fb2a5c88 (127.24.11.130:42221)
I20260928 06:31:22.182422 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:50085
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:22.201174 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:22.282881 30553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:22.283093 30553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:22.283118 30553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:22.284228 30553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:22.284266 30553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:22.284292 30553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:22.285822 30553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:35395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:22.286011 30553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:22.286186 30553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:22.291262 30553 server_base.cc:1034] running on GCE node
W20260928 06:31:22.291206 30559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:22.291196 30561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:22.291193 30558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:22.291720 30553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:22.292019 30553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:22.293217 30553 hybrid_clock.cc:648] HybridClock initialized: now 1790577082293201 us; error 35 us; skew 500 ppm
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.300203 30553 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:22.300736 30553 webserver.cc:460] Webserver started at http://127.24.11.131:39663/ using document root <none> and password file <none>
I20260928 06:31:22.301041 30553 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:22.301112 30553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:22.301257 30553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:22.302413 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "b0459f7306f54947b5bba8a9cf6df0d5"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "802700b40a26a55cf970e5fabca537c7"
server_key_iv: "62443435c29210126a05254fdd7701ca"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.302806 30553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "b0459f7306f54947b5bba8a9cf6df0d5"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "802700b40a26a55cf970e5fabca537c7"
server_key_iv: "62443435c29210126a05254fdd7701ca"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.304178 30553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:22.304965 30568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.305140 30553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.305244 30553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "b0459f7306f54947b5bba8a9cf6df0d5"
format_stamp: "Formatted at 2026-09-28 06:31:22 on dist-test-slave-38pg"
server_key: "802700b40a26a55cf970e5fabca537c7"
server_key_iv: "62443435c29210126a05254fdd7701ca"
server_key_version: "encryptionkey@0"
I20260928 06:31:22.305336 30553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:22.320358 30553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:22.322609 30553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:22.322748 30553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:22.322953 30553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:22.323241 30553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:22.323275 30553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.323303 30553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:22.323318 30553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:22.329330 30553 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:41921
I20260928 06:31:22.329416 30681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:41921 every 8 connection(s)
I20260928 06:31:22.329815 30553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:22.336262 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30553
I20260928 06:31:22.336390 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:22.336607 24622 external_mini_cluster.cc:1428] Setting key aa0d2a9e200c8f76d35acfd0968f1ded
I20260928 06:31:22.340142 30263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.330636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42405 (local address 127.24.11.190:35395)
0928 06:31:22.330701 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.330702 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.330936 (+ 234us) server_negotiation.cc:407] Connection header received
0928 06:31:22.331906 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.331907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.331930 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.331991 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.332720 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.333220 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.334323 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.334514 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.334566 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.337250 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.337251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.337252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.337259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.338850 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.339229 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.339230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.339230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.339281 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.339502 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.339503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.339503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.339604 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.339653 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.340003 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.340076 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:31:22.340296 30684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.330608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35395 (local address 127.24.11.131:42405)
0928 06:31:22.330783 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:22.330829 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:22.331812 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:22.332058 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:22.332062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:22.332381 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:22.332634 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.332639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.333326 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.333327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.334230 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:22.334232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.334585 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.334586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:22.334692 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.335292 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:22.335301 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:22.337152 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:22.338923 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.338929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.338931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.339162 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.339340 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:22.339341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:22.339341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:22.339446 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:22.339662 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:22.339666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:22.339833 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:31:22.339930 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:31:22.340062 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260928 06:31:22.340771 30682 heartbeater.cc:347] Connected to a master server at 127.24.11.190:35395
I20260928 06:31:22.340878 30682 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:22.341027 30682 heartbeater.cc:511] Master 127.24.11.190:35395 requested a full tablet report, sending...
I20260928 06:31:22.341411 30223 ts_manager.cc:194] Registered new tserver with Master: b0459f7306f54947b5bba8a9cf6df0d5 (127.24.11.131:41921)
I20260928 06:31:22.341833 30223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:42405
I20260928 06:31:22.348672 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/test-admin.keytab.
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.366127 24622 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:31:22.368011 30689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:22.367143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.24.11.129:41927)
0928 06:31:22.367335 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.367340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.367350 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:22.367375 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.367377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.367390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.367458 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.367929 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41552: BlockingRecv error: recv got EOF from 127.0.0.1:41552 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
W20260928 06:31:22.368096 24622 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.369148 24622 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.370242 24622 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:31:22.378301 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.370820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.24.11.129:41927)
0928 06:31:22.370883 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.370885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.370897 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:22.370968 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.370969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.370978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.371025 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.371405 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.372123 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.372853 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.373167 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.373222 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.375599 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.375608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.375609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.375615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.377151 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.377500 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.377503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.377503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.377610 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.377767 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.377768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.377768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.377904 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.378029 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.378156 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.378224 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:31:22.380282 30689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:22.379587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.24.11.129:41927)
0928 06:31:22.379644 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.379646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.379726 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:31:22.379767 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.379767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.379777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.379828 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.380217 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41572: BlockingRecv error: recv got EOF from 127.0.0.1:41572 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:31:22.380323 24622 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.381336 24622 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.382248 24622 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:31:22.389429 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.382825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.24.11.129:41927)
0928 06:31:22.382884 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.382892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.382946 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:31:22.382989 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.382990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.383000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.383055 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.383437 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.383938 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.384601 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.384857 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.384915 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.387011 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.387012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.387012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.387019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.388478 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.388768 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.388769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.388770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.388824 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.388982 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.388983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.388983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.389066 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.389132 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.389215 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.389288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 06:31:22.391052 30689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:22.390510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.24.11.129:41927)
0928 06:31:22.390546 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.390548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.390580 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:31:22.390630 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.390631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.390638 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.390683 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.390984 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41590: BlockingRecv error: recv got EOF from 127.0.0.1:41590 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:31:22.391130 24622 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577055356400-24622-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.392131 24622 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:22.392988 24622 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:31:22.399729 30689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:22.393463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.24.11.129:41927)
0928 06:31:22.393585 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:22.393587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:22.393614 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:31:22.393681 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:22.393681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:22.393689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:22.393735 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:22.394135 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.394593 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.395249 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:22.395420 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:22.395469 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:22.397593 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:22.397594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:22.397594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:22.397600 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:22.398778 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.399073 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.399074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.399075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.399122 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:22.399277 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:22.399278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:22.399278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:22.399360 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:22.399443 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:22.399583 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:31:22.399642 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:31:22.400691 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30285
I20260928 06:31:22.405907 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30419
I20260928 06:31:22.410519 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30553
I20260928 06:31:22.414518 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30191
2026-09-28T06:31:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1916 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30702](info): set up 2 sockets
Sep 28 06:31:22 dist-test-slave-38pg krb5kdc[30702](info): commencing operation
krb5kdc: starting...
W20260928 06:31:23.447029 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:23 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab.
Entry for principal oryx/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab.
I20260928 06:31:23.564841 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39467
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab
--principal=oryx/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:23.647493 30715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:23.647768 30715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:23.647807 30715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:23.649024 30715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:23.649103 30715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:23.649129 30715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:23.649152 30715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:23.649178 30715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:23.650811 30715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab
--principal=oryx/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:23.651114 30715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:23.651366 30715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:23.656998 30723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:23.657007 30721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:23.657074 30720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:23.657076 30715 server_base.cc:1034] running on GCE node
I20260928 06:31:23.657624 30715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:23.657941 30715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:23.659124 30715 hybrid_clock.cc:648] HybridClock initialized: now 1790577083659107 us; error 28 us; skew 500 ppm
Sep 28 06:31:23 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:23.661787 30715 init.cc:377] Logged in from keytab as oryx/127.24.11.190@KRBTEST.COM (short username oryx)
I20260928 06:31:23.662246 30715 webserver.cc:460] Webserver started at http://127.24.11.190:45177/ using document root <none> and password file <none>
I20260928 06:31:23.662456 30715 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:23.662496 30715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:23.662586 30715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:23.663635 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "04cb6c0b61a8482099c28c931c60bd03"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "2734fe8abda4c4f11bbf95eb2c313262"
server_key_iv: "e2fe71c199caa83e0734034732acd431"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.663961 30715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "04cb6c0b61a8482099c28c931c60bd03"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "2734fe8abda4c4f11bbf95eb2c313262"
server_key_iv: "e2fe71c199caa83e0734034732acd431"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.665290 30715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:31:23.666150 30730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:23.666349 30715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:23.666440 30715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "04cb6c0b61a8482099c28c931c60bd03"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "2734fe8abda4c4f11bbf95eb2c313262"
server_key_iv: "e2fe71c199caa83e0734034732acd431"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.666513 30715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:23.678649 30715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:23.681041 30715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:23.681160 30715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:23.685024 30715 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:39467
I20260928 06:31:23.685125 30782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:39467 every 8 connection(s)
I20260928 06:31:23.685439 30715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:23.686515 30783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:23.688808 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Bootstrap starting.
I20260928 06:31:23.689419 30783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:23.689777 30783 log.cc:824] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:23.690055 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30715
I20260928 06:31:23.690182 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:23.690357 24622 external_mini_cluster.cc:1428] Setting key 0d1ed4a0978eeedb3195bfc1061b1848
I20260928 06:31:23.690644 30783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: No bootstrap required, opened a new log
I20260928 06:31:23.692800 30783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } }
I20260928 06:31:23.692984 30783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:23.693020 30783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04cb6c0b61a8482099c28c931c60bd03, State: Initialized, Role: FOLLOWER
I20260928 06:31:23.693166 30783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } }
I20260928 06:31:23.693254 30783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:23.693287 30783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:23.693331 30783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:23.694116 30783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } }
I20260928 06:31:23.694229 30783 leader_election.cc:302] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04cb6c0b61a8482099c28c931c60bd03; no voters:
I20260928 06:31:23.694375 30783 leader_election.cc:288] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:23.694422 30788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:23.694571 30788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 LEADER]: Becoming Leader. State: Replica: 04cb6c0b61a8482099c28c931c60bd03, State: Running, Role: LEADER
I20260928 06:31:23.694614 30783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:23.694675 30788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } }
I20260928 06:31:23.695079 30790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } }
I20260928 06:31:23.695070 30788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04cb6c0b61a8482099c28c931c60bd03. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } }
I20260928 06:31:23.695163 30790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:23.695178 30788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
W20260928 06:31:23.696473 30803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:23.696532 30803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:23.696599 30796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:23.696803 30796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:23.698496 30796 catalog_manager.cc:1347] Generated new cluster ID: 658e86ed47074038a5755e21ea490458
I20260928 06:31:23.698552 30796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:23 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:23.702872 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.690937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.24.11.190:39467)
0928 06:31:23.691169 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:23.691179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:23.691215 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:31:23.692008 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:23.692018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:23.692083 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:23.692241 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:23.692835 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.694532 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.695692 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.696402 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.696517 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:23.699539 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:23.699543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:23.699543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:23.699556 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:23.701136 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:23.701510 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:23.701513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:23.701513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:23.701643 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:23.701851 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:23.701852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:23.701852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:23.702102 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:23.702266 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:23.702373 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:31:23.702526 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":128,"threads_started":1}
I20260928 06:31:23.713393 30796 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:23.714184 30796 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:23.722330 30796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Generated new TSK 0
I20260928 06:31:23.722579 30796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
I20260928 06:31:23.750829 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:23.839119 30809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:23.839353 30809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:23.839380 30809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:23.840592 30809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:23.840641 30809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:23.840669 30809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:23.842317 30809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:23.842604 30809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:23.842903 30809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:23.848150 30815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:23.848215 30814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:23.848273 30817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:23.848296 30809 server_base.cc:1034] running on GCE node
I20260928 06:31:23.848593 30809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:23.848865 30809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:23.850015 30809 hybrid_clock.cc:648] HybridClock initialized: now 1790577083850005 us; error 32 us; skew 500 ppm
Sep 28 06:31:23 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:23.852788 30809 init.cc:377] Logged in from keytab as oryx/127.24.11.129@KRBTEST.COM (short username oryx)
I20260928 06:31:23.853255 30809 webserver.cc:460] Webserver started at http://127.24.11.129:36853/ using document root <none> and password file <none>
I20260928 06:31:23.853498 30809 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:23.853598 30809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:23.853755 30809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:23.854928 30809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "e1fb529453174b2fb2ad5e088ef7e246"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "b36334a0035e25aeb6dc1d199faf11dc"
server_key_iv: "b69ce42442d572e9aa8a6c9dd7d5c035"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.855298 30809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "e1fb529453174b2fb2ad5e088ef7e246"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "b36334a0035e25aeb6dc1d199faf11dc"
server_key_iv: "b69ce42442d572e9aa8a6c9dd7d5c035"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.856823 30809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:23.857823 30824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:23.858053 30809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:23.858166 30809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "e1fb529453174b2fb2ad5e088ef7e246"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "b36334a0035e25aeb6dc1d199faf11dc"
server_key_iv: "b69ce42442d572e9aa8a6c9dd7d5c035"
server_key_version: "encryptionkey@0"
I20260928 06:31:23.858242 30809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:23.868709 30809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:23.871325 30809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:23.871505 30809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:23.871760 30809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:23.872115 30809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:23.872180 30809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:23.872231 30809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:23.872262 30809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:23.878187 30809 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:36191
I20260928 06:31:23.878252 30937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:36191 every 8 connection(s)
I20260928 06:31:23.878639 30809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:31:23 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:23.886217 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30809
I20260928 06:31:23.886338 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:23.886567 24622 external_mini_cluster.cc:1428] Setting key 99491e8a29740f849cf63733b5853bf6
I20260928 06:31:23.888350 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.879482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:54981 (local address 127.24.11.190:39467)
0928 06:31:23.879528 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:23.879530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:23.879737 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:31:23.880220 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:23.880221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:23.880231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:23.880291 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:23.881217 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.881737 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.882784 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.883014 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.883070 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:23.885297 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:23.885299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:23.885299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:23.885318 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:23.886879 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:23.887333 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:23.887334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:23.887334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:23.887396 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:23.887692 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:23.887693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:23.887693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:23.887796 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:23.887883 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:23.888216 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 06:31:23.888285 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:23.888672 30940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:23.879434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:54981)
0928 06:31:23.879598 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:23.879642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:23.880133 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:23.880357 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:23.880360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:23.880939 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:23.881135 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:23.881140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.881870 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.881871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:23.882685 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:23.882687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:23.883087 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:23.883088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:23.883193 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:23.883675 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:23.883683 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:23.885204 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:23.886952 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:23.886959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:23.886962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:23.887262 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:23.887471 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:23.887471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:23.887472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:23.887626 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:23.887894 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:23.887900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:23.888078 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 06:31:23.888197 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:31:23.888383 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260928 06:31:23.889264 30938 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:23.889356 30938 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:23.889598 30938 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:23.890161 30747 ts_manager.cc:194] Registered new tserver with Master: e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
I20260928 06:31:23.890790 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:54981
WARNING: no policy specified for oryx/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
I20260928 06:31:23.913472 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:23.996819 30943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:23.997045 30943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:23.997071 30943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:23.998314 30943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:23.998363 30943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:23.998391 30943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:24.000025 30943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:24.000305 30943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.000551 30943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.005622 30949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.005693 30943 server_base.cc:1034] running on GCE node
W20260928 06:31:24.005692 30948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.005617 30951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.006186 30943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.006469 30943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.007622 30943 hybrid_clock.cc:648] HybridClock initialized: now 1790577084007586 us; error 52 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.010314 30943 init.cc:377] Logged in from keytab as oryx/127.24.11.130@KRBTEST.COM (short username oryx)
I20260928 06:31:24.010793 30943 webserver.cc:460] Webserver started at http://127.24.11.130:45247/ using document root <none> and password file <none>
I20260928 06:31:24.011063 30943 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.011111 30943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.011276 30943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:24.012342 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "b1f7387082dc418c8f1d39e8f51d6086"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "8bca9db05bfa657100ec94761d726daf"
server_key_iv: "9b090127c9f83e97e875e8e3ebb479b9"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.012705 30943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "b1f7387082dc418c8f1d39e8f51d6086"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "8bca9db05bfa657100ec94761d726daf"
server_key_iv: "9b090127c9f83e97e875e8e3ebb479b9"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.014146 30943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:24.014999 30958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.015209 30943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:24.015282 30943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "b1f7387082dc418c8f1d39e8f51d6086"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "8bca9db05bfa657100ec94761d726daf"
server_key_iv: "9b090127c9f83e97e875e8e3ebb479b9"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.015375 30943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.026889 30943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.028869 30943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:24.029021 30943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.029284 30943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:24.029692 30943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:24.029731 30943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.029754 30943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:24.029784 30943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.035784 30943 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:43413
I20260928 06:31:24.035867 31071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:43413 every 8 connection(s)
I20260928 06:31:24.036170 30943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:24.038477 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 30943
I20260928 06:31:24.038590 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:24.038791 24622 external_mini_cluster.cc:1428] Setting key a1e0b79a71d04f5b2ac6be5c37584785
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:24.047451 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.036935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:45253 (local address 127.24.11.190:39467)
0928 06:31:24.037014 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.037017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.037217 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:31:24.037838 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.037839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.037863 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.037933 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.039273 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.040033 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.041062 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.041390 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.041469 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.044010 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.044011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.044012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.044021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.045931 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.046371 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.046372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.046373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.046432 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.046677 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.046678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.046679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.046843 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.046929 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.047239 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.047339 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:24.047546 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.036860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:45253)
0928 06:31:24.037075 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.037125 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.037671 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.038011 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.038015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.038846 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.039168 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.039175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.040149 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.040150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.040966 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.040969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.041500 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.041501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.041687 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.042333 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.042342 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.043896 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.046033 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.046040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.046042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.046288 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.046506 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.046506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.046507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.046610 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.046943 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.046950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.047084 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.047186 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.047326 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.24.11.131@KRBTEST.COM; defaulting to no policy
I20260928 06:31:24.048149 31072 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:24.048239 31072 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:24.048394 31072 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:24.048816 30747 ts_manager.cc:194] Registered new tserver with Master: b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413)
I20260928 06:31:24.049254 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:45253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
I20260928 06:31:24.064749 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:24.147210 31077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:24.147446 31077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:24.147471 31077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:24.148690 31077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:24.148737 31077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:24.148765 31077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:24.150378 31077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:39467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:24.150579 31077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.150763 31077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.155913 31082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.155921 31083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.155932 31085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.156075 31077 server_base.cc:1034] running on GCE node
I20260928 06:31:24.156494 31077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.156713 31077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.157872 31077 hybrid_clock.cc:648] HybridClock initialized: now 1790577084157836 us; error 32 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.160619 31077 init.cc:377] Logged in from keytab as oryx/127.24.11.131@KRBTEST.COM (short username oryx)
I20260928 06:31:24.161036 31077 webserver.cc:460] Webserver started at http://127.24.11.131:38639/ using document root <none> and password file <none>
I20260928 06:31:24.161238 31077 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.161329 31077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.161460 31077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:24.162555 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "ee6ab8b7e6ef7fc8c8c48855e55ed0c0"
server_key_iv: "fcf12d1899ac54312656722e3eb11b8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.162909 31077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "ee6ab8b7e6ef7fc8c8c48855e55ed0c0"
server_key_iv: "fcf12d1899ac54312656722e3eb11b8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.164346 31077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:24.165112 31092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.165318 31077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:24.165433 31077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "ee6ab8b7e6ef7fc8c8c48855e55ed0c0"
server_key_iv: "fcf12d1899ac54312656722e3eb11b8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.165510 31077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.181262 31077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.183414 31077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:24.183585 31077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.183821 31077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:24.184139 31077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:24.184191 31077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.184233 31077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:24.184259 31077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.189900 31077 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:39971
I20260928 06:31:24.190011 31205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:39971 every 8 connection(s)
I20260928 06:31:24.190308 31077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:24.200250 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31077
I20260928 06:31:24.200481 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:24.200613 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.191080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:41829 (local address 127.24.11.190:39467)
0928 06:31:24.191145 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.191147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.191354 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:31:24.191951 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.191953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.191966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.192019 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.193004 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.193576 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.194701 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.195004 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.195095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.197500 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.197502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.197502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.197510 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.199053 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.199497 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.199498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.199498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.199564 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.199884 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.199885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.199886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.200022 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.200139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.200432 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.200519 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:24.200695 24622 external_mini_cluster.cc:1428] Setting key c440929dccc555e2e2eea27fcf74faea
I20260928 06:31:24.200832 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.191046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:41829)
0928 06:31:24.191234 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.191271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.191781 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.192091 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.192095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.192697 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.192900 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.192905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.193662 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.193663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.194605 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.194607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.195113 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.195114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.195235 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.195824 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.195833 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.197385 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.199161 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.199169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.199171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.199409 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.199631 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.199632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.199632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.199823 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.200132 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.200139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.200291 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.200408 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.200576 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260928 06:31:24.201470 31206 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:24.201601 31206 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:24.201825 31206 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:24.202221 30747 ts_manager.cc:194] Registered new tserver with Master: 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:24.202595 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:41829
I20260928 06:31:24.212873 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.11.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.189.keytab.
Entry for principal oryx/127.24.11.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.189.keytab.
I20260928 06:31:24.240358 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:34195
--webserver_interface=127.24.11.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.189.keytab
--principal=oryx/127.24.11.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39467,127.24.11.189:34195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:24.322685 31211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:24.322906 31211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:24.322929 31211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:24.324038 31211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:24.324071 31211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:24.324086 31211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:24.324098 31211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:24.324110 31211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:24.325500 31211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39467,127.24.11.189:34195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.189.keytab
--principal=oryx/127.24.11.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:34195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.11.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
W20260928 06:31:24.325624 31211 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.11.190:39467,127.24.11.189:34195'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:31:24.325736 31211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.325906 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.330778 31219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.330780 31217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.330832 31211 server_base.cc:1034] running on GCE node
W20260928 06:31:24.330835 31216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.331315 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.331604 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.332790 31211 hybrid_clock.cc:648] HybridClock initialized: now 1790577084332764 us; error 51 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.335458 31211 init.cc:377] Logged in from keytab as oryx/127.24.11.189@KRBTEST.COM (short username oryx)
I20260928 06:31:24.335968 31211 webserver.cc:460] Webserver started at http://127.24.11.189:43673/ using document root <none> and password file <none>
I20260928 06:31:24.336198 31211 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.336267 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.336426 31211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:24.337632 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/instance:
uuid: "3b545cd2ded54018a84389c63c1e4103"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "2b90b39d83535c998f2f23caad5c4657"
server_key_iv: "92dc32206387b834310e820836e0eea2"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.338047 31211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal/instance:
uuid: "3b545cd2ded54018a84389c63c1e4103"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "2b90b39d83535c998f2f23caad5c4657"
server_key_iv: "92dc32206387b834310e820836e0eea2"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.339433 31211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:24.340173 31226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.340342 31211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:24.340459 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "3b545cd2ded54018a84389c63c1e4103"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "2b90b39d83535c998f2f23caad5c4657"
server_key_iv: "92dc32206387b834310e820836e0eea2"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.340541 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.357402 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.359997 31211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:24.360168 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.364490 31211 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.189:34195
I20260928 06:31:24.364529 31278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.189:34195 every 8 connection(s)
I20260928 06:31:24.364913 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
I20260928 06:31:24.365425 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31211
I20260928 06:31:24.366170 31279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:24.367991 31279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.189@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:24.376772 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.368705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.24.11.190:39467)
0928 06:31:24.368766 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.368768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.368923 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:31:24.369007 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.369008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.369025 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.369107 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.369706 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.370247 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.371087 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.371414 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.371500 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.373868 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.373869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.373869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.373878 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.375396 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.375803 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.375804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.375804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.375863 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.376104 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.376105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.376106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.376289 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.376391 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.376590 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.376675 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:31:24.377048 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.368650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37380)
0928 06:31:24.368819 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.368851 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.368943 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.369120 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.369122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.369518 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.369623 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.369625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.370377 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.370377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.370993 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.370995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.371505 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.371506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.371557 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.372087 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.372098 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.373773 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.375472 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.375478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.375480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.375737 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.375930 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.375931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.375931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.376032 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.376387 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.376393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.376460 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.376558 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.376795 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":72,"thread_start_us":76,"threads_started":1}
I20260928 06:31:24.377046 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.366169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.24.11.189:34195)
0928 06:31:24.366402 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.366411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.366436 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:24.366864 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.366871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.366917 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.367070 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.367667 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.368820 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.369693 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.370124 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.370246 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.373235 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.373242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.373242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.373248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.375278 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.375693 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.375696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.375696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.375800 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.376029 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.376029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.376029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.376317 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.376483 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.376574 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.376700 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260928 06:31:24.377887 31279 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.189@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
I20260928 06:31:24.383846 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.378132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:59968)
0928 06:31:24.378178 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.378189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.378227 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.378343 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.378344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.378479 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.378511 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.378512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.379133 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.379134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.379698 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.379700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.380055 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.380056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.380120 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.380475 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.380479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.381615 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.382883 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.382884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.382884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.383045 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.383244 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.383245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.383245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.383298 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.383538 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.383539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.383575 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.383642 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.383737 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:31:24.383896 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.378198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.24.11.189:34195)
0928 06:31:24.378232 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.378234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.378241 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:24.378280 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.378280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.378289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.378340 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.378582 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.379048 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.379784 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.379990 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.380061 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.381715 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.381716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.381716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.381723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.382798 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.383128 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.383129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.383129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.383182 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.383371 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.383372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.383372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.383475 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.383547 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.383667 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.383783 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":19}
I20260928 06:31:24.385255 31279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Bootstrap starting.
I20260928 06:31:24.386188 31279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:24.386471 31279 log.cc:824] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:24.387261 31279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: No bootstrap required, opened a new log
I20260928 06:31:24.388667 31279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } }
I20260928 06:31:24.388855 31279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:24.388926 31279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b545cd2ded54018a84389c63c1e4103, State: Initialized, Role: FOLLOWER
I20260928 06:31:24.389078 31279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } }
I20260928 06:31:24.389349 31285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } } }
I20260928 06:31:24.389411 31279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:24.389428 31285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:31:24.391192 31296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:24.391289 31296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:24.392825 31211 master_runner.cc:428] Detected that this master 3b545cd2ded54018a84389c63c1e4103 is joining an existing cluster
I20260928 06:31:24.392884 31211 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.11.189:34195
I20260928 06:31:24.400373 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.394178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.24.11.189:34195)
0928 06:31:24.394219 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.394221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.394374 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:31:24.394469 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.394470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.394478 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.394524 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.394951 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.395515 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.396245 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.396498 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.396575 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.397604 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.397606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.397606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.397612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.399202 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.399575 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.399576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.399576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.399624 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.399797 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.399798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.399798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.399912 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.399951 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.400145 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.400268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:24.400518 31302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.394152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:59976)
0928 06:31:24.394295 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.394315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.394397 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.394560 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.394560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.394782 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.394877 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.394880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.395581 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.395582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.396151 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.396152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.396589 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.396590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.396645 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.397133 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.397138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.397463 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.399288 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.399288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.399289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.399499 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.399686 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.399686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.399687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.399737 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.399989 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.399990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.400022 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.400241 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.400366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260928 06:31:24.401506 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.393976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.24.11.190:39467)
0928 06:31:24.394033 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.394036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.394363 (+ 327us) server_negotiation.cc:407] Connection header received
0928 06:31:24.394440 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.394441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.394452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.394506 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.395011 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.395798 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.396887 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.397180 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.397221 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.398549 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.398550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.398550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.398557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.399948 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.400442 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.400443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.400443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.400508 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.400716 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.400718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.400718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.400863 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.400942 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.401284 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.401406 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:31:24.402021 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.393891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37396)
0928 06:31:24.394279 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.394297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.394376 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.394509 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.394510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.394807 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.394917 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.394919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.395909 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.395910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.396787 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.396790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.397294 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.397295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.397369 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.397878 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.397883 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.398243 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.400048 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.400049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.400050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.400347 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.400589 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.400590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.400590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.400643 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.400956 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.400957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.401029 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.401620 (+ 591us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.401734 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":106,"threads_started":1}
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.190@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
I20260928 06:31:24.411811 31282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.404305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.24.11.189:34195)
0928 06:31:24.404339 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.404341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.404514 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:31:24.404610 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.404611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.404618 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.404664 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.405146 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.405674 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.406811 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.407016 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.407080 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.409244 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.409245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.409246 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.409253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.410619 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.411007 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.411008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.411008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.411066 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.411266 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.411266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.411267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.411378 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.411449 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.411630 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.411709 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:24.411841 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.404261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:59982)
0928 06:31:24.404429 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.404469 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.404538 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.404669 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.404672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.405011 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.405071 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.405074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.405746 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.405747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.406707 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.406709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.407095 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.407096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.407182 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.407655 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.407660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.409154 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.410705 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.410713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.410716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.410930 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.411134 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.411135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.411136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.411204 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.411454 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.411462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.411513 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.411606 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.411726 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":67,"threads_started":1}
I20260928 06:31:24.412628 30747 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.11.189:34195: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3b545cd2ded54018a84389c63c1e4103"
member_type: NON_VOTER
last_known_addr {
host: "127.24.11.189"
port: 34195
}
attrs {
promote: true
}
}
dest_uuid: "04cb6c0b61a8482099c28c931c60bd03"
cas_config_opid_index: -1
I20260928 06:31:24.412966 30747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } }
I20260928 06:31:24.413462 31306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3b545cd2ded54018a84389c63c1e4103 (127.24.11.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } }
I20260928 06:31:24.413793 31253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:24.414448 31306 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.11.189:34195
I20260928 06:31:24.414538 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } }
I20260928 06:31:24.414626 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:24.414757 31253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Refusing update from remote peer 04cb6c0b61a8482099c28c931c60bd03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:31:24.415088 31211 master.cc:537] Master@127.24.11.189:34195 shutting down...
I20260928 06:31:24.415108 31305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:31:24.415874 30732 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:31:24.416458 30732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 -> Peer 3b545cd2ded54018a84389c63c1e4103 (127.24.11.189:34195): Couldn't send request to peer 3b545cd2ded54018a84389c63c1e4103. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:31:24.417656 31211 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:24.417739 31211 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:24.417804 31211 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: stopping tablet replica
I20260928 06:31:24.430649 31211 master.cc:552] Master@127.24.11.189:34195 shutdown complete.
I20260928 06:31:24.430713 31211 master_runner.cc:305] Clearing existing system tablet
I20260928 06:31:24.430847 31211 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.430897 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.431674 31211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:24.432206 31310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.432371 31211 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.432436 31211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.432461 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "3b545cd2ded54018a84389c63c1e4103"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "2b90b39d83535c998f2f23caad5c4657"
server_key_iv: "92dc32206387b834310e820836e0eea2"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.433772 31211 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:24.435350 31211 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:31:24.435420 31211 log.cc:1196] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:31:24.435609 31211 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Deleting consensus metadata
I20260928 06:31:24.435731 31211 master_runner.cc:315] Copying system tablet from 127.24.11.190:39467
I20260928 06:31:24.436343 31211 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: tablet copy: Beginning tablet copy session from remote peer at address 127.24.11.190:39467
I20260928 06:31:24.442818 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.436728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.24.11.190:39467)
0928 06:31:24.436778 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.436780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.436822 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:31:24.436901 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.436901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.436910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.436960 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.437307 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.437871 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.438934 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.439117 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.439182 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.440206 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.440207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.440207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.440214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.441752 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.442116 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.442117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.442118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.442167 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.442372 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.442372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.442373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.442470 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.442519 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.442682 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.442739 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:24.442863 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.436634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37408)
0928 06:31:24.436753 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.436764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.436838 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.436967 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.436968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.437174 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.437237 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.437239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.437945 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.437946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.438832 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.438835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.439194 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.439195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.439261 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.439727 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.439732 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.440121 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.441827 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.441828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.441829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.442048 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.442234 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.442235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.442236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.442312 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.442530 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.442531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.442577 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.442660 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.442761 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
I20260928 06:31:24.443488 30767 tablet_copy_service.cc:140] P 04cb6c0b61a8482099c28c931c60bd03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3b545cd2ded54018a84389c63c1e4103 ({username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:37408)
I20260928 06:31:24.443571 30767 tablet_copy_service.cc:161] P 04cb6c0b61a8482099c28c931c60bd03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3b545cd2ded54018a84389c63c1e4103 at {username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:37408: session id = 3b545cd2ded54018a84389c63c1e4103-00000000000000000000000000000000
I20260928 06:31:24.444190 30767 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:31:24.444829 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:31:24.446126 31211 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: tablet copy: Starting download of 0 data blocks...
I20260928 06:31:24.446278 31211 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: tablet copy: Starting download of 1 WAL segments...
I20260928 06:31:24.447150 31211 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:31:24.448115 30767 tablet_copy_service.cc:342] P 04cb6c0b61a8482099c28c931c60bd03: Request end of tablet copy session 3b545cd2ded54018a84389c63c1e4103-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:37408
I20260928 06:31:24.448172 30767 tablet_copy_service.cc:433] P 04cb6c0b61a8482099c28c931c60bd03: ending tablet copy session 3b545cd2ded54018a84389c63c1e4103-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3b545cd2ded54018a84389c63c1e4103
I20260928 06:31:24.450493 31211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.451705 31317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.451716 31318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.451716 31320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.451954 31211 server_base.cc:1034] running on GCE node
I20260928 06:31:24.452029 31211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.452159 31211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.453280 31211 hybrid_clock.cc:648] HybridClock initialized: now 1790577084453269 us; error 20 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.454620 31211 init.cc:377] Logged in from keytab as oryx/127.24.11.189@KRBTEST.COM (short username oryx)
I20260928 06:31:24.455044 31211 webserver.cc:460] Webserver started at http://127.24.11.189:32877/ using document root <none> and password file <none>
I20260928 06:31:24.455188 31211 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.455231 31211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.455862 31211 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:24.456389 31327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.456509 31211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:24.456555 31211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "3b545cd2ded54018a84389c63c1e4103"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "2b90b39d83535c998f2f23caad5c4657"
server_key_iv: "92dc32206387b834310e820836e0eea2"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.456610 31211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.472362 31211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.474072 31211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.477200 31211 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.189:34195
I20260928 06:31:24.477397 31379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.189:34195 every 8 connection(s)
I20260928 06:31:24.477600 31211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
I20260928 06:31:24.478310 31380 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:24.478842 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Bootstrap starting.
I20260928 06:31:24.481431 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:24.481712 31380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Bootstrap complete.
I20260928 06:31:24.481932 31380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } }
I20260928 06:31:24.482019 31380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3b545cd2ded54018a84389c63c1e4103, State: Initialized, Role: LEARNER
I20260928 06:31:24.482052 31380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } }
I20260928 06:31:24.482201 31380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:24.483510 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } }
I20260928 06:31:24.483613 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:31:24.483805 31395 catalog_manager.cc:1259] Loaded cluster ID: 658e86ed47074038a5755e21ea490458
I20260928 06:31:24.483945 31395 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: loading cluster ID for follower catalog manager: success
I20260928 06:31:24.485321 31395 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: acquiring CA information for follower catalog manager: success
I20260928 06:31:24.485719 31395 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:31:24.487255 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.478836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.24.11.189:34195)
0928 06:31:24.479037 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.479040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.479048 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:24.479070 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.479071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.479082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.479154 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.479866 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.480841 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.481899 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.482198 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.482543 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.483850 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.483851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.483851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.483858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.486092 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.486456 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.486457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.486457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.486514 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.486722 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.486723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.486724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.486836 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.486938 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.487067 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.487157 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.24.11.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.188.keytab.
Entry for principal oryx/127.24.11.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.188.keytab.
I20260928 06:31:24.514336 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:36087
--webserver_interface=127.24.11.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.188.keytab
--principal=oryx/127.24.11.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:24.608978 31398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:24.609257 31398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:24.609282 31398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:24.610474 31398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:24.610517 31398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:24.610531 31398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:24.610544 31398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:24.610556 31398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:24.612077 31398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.188.keytab
--principal=oryx/127.24.11.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.11.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:24.612329 31398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.612584 31398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.617415 31406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.617412 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.617453 31398 server_base.cc:1034] running on GCE node
W20260928 06:31:24.617449 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.617898 31398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.618261 31398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.619402 31398 hybrid_clock.cc:648] HybridClock initialized: now 1790577084619389 us; error 28 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.621824 31398 init.cc:377] Logged in from keytab as oryx/127.24.11.188@KRBTEST.COM (short username oryx)
I20260928 06:31:24.622279 31398 webserver.cc:460] Webserver started at http://127.24.11.188:45389/ using document root <none> and password file <none>
I20260928 06:31:24.622484 31398 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.622525 31398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.622615 31398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:24.623618 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/instance:
uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "bec1413bcdb9ed9819ec05b62470df53"
server_key_iv: "0b72aa930c42bf8c9df8f7420e3be6c3"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.623924 31398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal/instance:
uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "bec1413bcdb9ed9819ec05b62470df53"
server_key_iv: "0b72aa930c42bf8c9df8f7420e3be6c3"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.625197 31398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:31:24.625989 31413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.626176 31398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:24.626241 31398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "bec1413bcdb9ed9819ec05b62470df53"
server_key_iv: "0b72aa930c42bf8c9df8f7420e3be6c3"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.626294 31398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.648074 31398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.650444 31398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:24.650568 31398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.655026 31398 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.188:36087
I20260928 06:31:24.655193 31465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.188:36087 every 8 connection(s)
I20260928 06:31:24.655444 31398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
I20260928 06:31:24.656643 31466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:24.658401 31466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:31:24.660029 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31398
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.188@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
I20260928 06:31:24.669708 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.659231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.24.11.190:39467)
0928 06:31:24.659347 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.659349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.659436 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:31:24.659942 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.659944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.659959 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.660061 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.661070 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.661804 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.662749 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.662959 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.663047 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.666074 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.666075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.666076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.666084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.667890 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.668513 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.668513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.668514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.668583 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.668876 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.668877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.668878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.669049 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.669122 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.669490 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.669603 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 06:31:24.669952 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.660634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.24.11.188:36087)
0928 06:31:24.660830 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.660835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.660854 (+ 19us) server_negotiation.cc:407] Connection header received
0928 06:31:24.660878 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.660881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.660900 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.660989 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.661755 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.662580 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.663311 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.663637 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.663715 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.666133 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.666139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.666140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.666146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.668278 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.668700 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.668703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.668704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.668802 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.669006 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.669007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.669007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.669287 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.669444 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.669588 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.669722 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260928 06:31:24.669958 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.659081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37414)
0928 06:31:24.659270 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.659314 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.659814 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.660265 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.660270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.660669 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.660995 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.661000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.661911 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.661912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.662651 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.662653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.663076 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.663078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.663207 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.663819 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.663829 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.665966 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.667997 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.668003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.668005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.668407 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.668672 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.668672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.668673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.668799 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.669141 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.669147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.669292 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.669419 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.669647 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260928 06:31:24.670596 31466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.188@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
I20260928 06:31:24.676978 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.670852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:60006)
0928 06:31:24.670901 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.670912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.670947 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.671086 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.671087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.671196 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.671232 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.671233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.671831 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.671831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.672327 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.672328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.672720 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.672721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.672782 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.673259 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.673264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.674485 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.675967 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.675968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.675968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.676133 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.676348 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.676349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.676349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.676423 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.676718 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.676719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.676762 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.676821 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.676907 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:31:24.677016 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.670937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.24.11.189:34195)
0928 06:31:24.670983 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.670985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.670993 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:24.671015 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.671016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.671032 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.671082 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.671273 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.671744 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.672429 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.672631 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.672697 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.674573 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.674575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.674575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.674581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.675876 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.676203 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.676205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.676205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.676259 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.676512 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.676513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.676513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.676645 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.676707 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.676879 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.676951 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:24.677479 31466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.188@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
I20260928 06:31:24.683799 31468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.677763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46322)
0928 06:31:24.677805 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.677816 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.677856 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.677996 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.677997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.678108 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.678140 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.678141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.678701 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.678702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.679206 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.679207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.679632 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.679632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.679669 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.680118 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.680131 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.681251 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.682694 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.682695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.682695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.682898 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.683061 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.683062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.683062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.683111 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.683424 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.683424 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.683462 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.683548 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.683644 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 06:31:24.683859 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.677822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.24.11.188:36087)
0928 06:31:24.677859 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.677863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.677876 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:24.677946 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.677946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.677954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.677993 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.678178 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.678631 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.679302 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.679561 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.679628 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.681325 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.681327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.681327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.681333 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.682624 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.682961 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.682961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.682962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.683007 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.683206 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.683207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.683207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.683335 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.683430 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.683567 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.683714 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":18}
I20260928 06:31:24.685385 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Bootstrap starting.
I20260928 06:31:24.686431 31466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:24.686738 31466 log.cc:824] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:24.687568 31466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: No bootstrap required, opened a new log
I20260928 06:31:24.689152 31466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } }
I20260928 06:31:24.689273 31466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:24.689296 31466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2757720e6d46f3a6330cb4b1bd2f6e, State: Initialized, Role: FOLLOWER
I20260928 06:31:24.689391 31466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } }
I20260928 06:31:24.689700 31466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:24.689659 31471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } } }
I20260928 06:31:24.689774 31471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:31:24.691350 31482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:24.691423 31482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:24.693778 31398 master_runner.cc:428] Detected that this master 7f2757720e6d46f3a6330cb4b1bd2f6e is joining an existing cluster
I20260928 06:31:24.693840 31398 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.11.188:36087
I20260928 06:31:24.702226 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.695450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.24.11.188:36087)
0928 06:31:24.695496 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.695498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.695630 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:31:24.695752 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.695753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.695763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.695790 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.696183 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.696915 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698046 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.698319 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698359 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.699518 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.699519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.699519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.699526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.701012 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.701378 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.701379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.701379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.701431 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.701680 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.701681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.701681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.701789 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.701862 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.702020 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.702121 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":16}
I20260928 06:31:24.702420 31490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.695333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46336)
0928 06:31:24.695577 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.695586 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.695651 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.695841 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.695842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.696040 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.696106 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.696107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.697034 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.697035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.697958 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.697961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.698438 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.698490 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.698982 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.698987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.699428 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.701106 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.701107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.701107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.701286 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.701501 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.701502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.701502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.701600 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.701852 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.701853 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.701911 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.702000 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.702173 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":79,"thread_start_us":58,"threads_started":1}
I20260928 06:31:24.721318 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30809
I20260928 06:31:24.722731 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.694947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37430)
0928 06:31:24.695098 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.695109 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.695171 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.695332 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.695332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.695663 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.695781 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.695783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.697148 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.697148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.697989 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.697991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.698543 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.698613 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.718605 (+ 19992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.718610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.719101 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.720928 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.720929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.720929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.721134 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.721328 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.721328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.721328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.721393 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.722294 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.722295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.722345 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.722417 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.722562 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260928 06:31:24.723085 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.694945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.24.11.190:39467)
0928 06:31:24.694990 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.694992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.695214 (+ 222us) server_negotiation.cc:407] Connection header received
0928 06:31:24.695241 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.695242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.695254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.695326 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.695854 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.696651 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698068 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.698381 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698485 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.719197 (+ 20712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.719199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.719199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.719208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.720844 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.721211 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.721212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.721212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.721262 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.721461 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.721462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.721462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.722185 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.722266 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.722854 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.722944 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:24.724014 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.695378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.24.11.189:34195)
0928 06:31:24.695416 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.695418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.695426 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:24.695953 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.695954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.695964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.696010 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.696407 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.697085 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.699315 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.699610 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.699686 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.718851 (+ 19165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.718853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.718853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.718862 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.720489 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.720928 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.720929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.720929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.720985 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.721192 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.721193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.721194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.721347 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.721391 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.723842 (+ 2451us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.723925 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:24.724040 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.695164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:60020)
0928 06:31:24.695266 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.695275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.695873 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.696102 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.696103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.696264 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.696312 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.696313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.698086 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.698086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.699225 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.699228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.699693 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.699694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.699767 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.718224 (+ 18457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.718232 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.718731 (+ 499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.720584 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.720586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.720586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.720810 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.721061 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.721061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.721061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.721120 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.723690 (+ 2570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.723691 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.723729 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.723812 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.723926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 06:31:24.727995 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:36191
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:39467
--webserver_port=36853
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577083, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.190@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
I20260928 06:31:24.736454 31469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.729660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.24.11.188:36087)
0928 06:31:24.729700 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.729702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.729708 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:31:24.729737 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.729737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.729746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.729792 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.730177 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.730909 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.731617 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.731824 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.731896 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.733810 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.733811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.733812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.733818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.735304 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.735667 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.735668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.735669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.735729 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.735931 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.735933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.735933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.736050 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.736124 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.736282 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.736360 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:24.736466 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.729390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46348)
0928 06:31:24.729467 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.729476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.729511 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.729850 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.729851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.730049 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.730099 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.730101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.730999 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.731000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.731521 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.731523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.731906 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.731907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.732005 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.732456 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.732462 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.733703 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.735407 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.735408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.735409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.735586 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.735797 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.735798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.735799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.735859 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.736116 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.736117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.736163 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.736251 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.736361 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 06:31:24.737275 30747 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.11.188:36087: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
member_type: NON_VOTER
last_known_addr {
host: "127.24.11.188"
port: 36087
}
attrs {
promote: true
}
}
dest_uuid: "04cb6c0b61a8482099c28c931c60bd03"
cas_config_opid_index: 5
I20260928 06:31:24.737581 30747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } }
I20260928 06:31:24.737972 31306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.738758 31306 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.11.188:36087
I20260928 06:31:24.738847 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.738943 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:24.739143 31398 master.cc:537] Master@127.24.11.188:36087 shutting down...
W20260928 06:31:24.740257 30732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 -> Peer 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188:36087): Couldn't send request to peer 7f2757720e6d46f3a6330cb4b1bd2f6e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:31:24.742375 31398 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:24.742465 31398 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:24.742504 31398 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: stopping tablet replica
I20260928 06:31:24.744236 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.740350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.24.11.189:34195)
0928 06:31:24.740394 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.740396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.740475 (+ 79us) server_negotiation.cc:407] Connection header received
0928 06:31:24.740576 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.740577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.740602 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.740644 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:24.740859 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.741632 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.743122 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.743746 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.743817 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.743903 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.744115 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.744162 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:31:24.744645 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.739216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:60024)
0928 06:31:24.740423 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.740435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.740499 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.740719 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.740720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.740722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:24.740777 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.740779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.741736 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.741736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.742953 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.742956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.743937 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.743938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.744007 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.744028 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.744509 (+ 481us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.744557 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1191}
I20260928 06:31:24.745245 31354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 LEARNER]: Refusing update from remote peer 04cb6c0b61a8482099c28c931c60bd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:31:24.745453 31305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:31:24.746217 31354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.747225 31493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.747321 31493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:31:24.747417 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04cb6c0b61a8482099c28c931c60bd03. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: NON_VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: true } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.747470 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:31:24.747704 31305 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: attempting to promote NON_VOTER 3b545cd2ded54018a84389c63c1e4103 to VOTER
I20260928 06:31:24.747933 31305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } }
I20260928 06:31:24.748508 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.748374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46356)
0928 06:31:24.748428 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.748477 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:31:24.748550 31354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 LEARNER]: Refusing update from remote peer 04cb6c0b61a8482099c28c931c60bd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:31:24.748745 31494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:31:24.748910 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.748827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46370)
0928 06:31:24.748864 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.748885 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:31:24.749089 30732 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 -> Peer 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188:36087): Couldn't send request to peer 7f2757720e6d46f3a6330cb4b1bd2f6e. Status: Network error: Client connection negotiation failed: client connection to 127.24.11.188:36087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:31:24.749482 31306 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b545cd2ded54018a84389c63c1e4103 (127.24.11.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.750260 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.750371 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:24.750432 31306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.750481 31306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:24.750695 31354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3b545cd2ded54018a84389c63c1e4103 (127.24.11.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.751447 31384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.751533 31384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:24.755923 31398 master.cc:552] Master@127.24.11.188:36087 shutdown complete.
I20260928 06:31:24.755993 31398 master_runner.cc:305] Clearing existing system tablet
I20260928 06:31:24.756126 31398 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.756187 31398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.756927 31398 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:24.757442 31496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.757586 31398 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.757647 31398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.757675 31398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "bec1413bcdb9ed9819ec05b62470df53"
server_key_iv: "0b72aa930c42bf8c9df8f7420e3be6c3"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.758374 31398 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:24.759841 31398 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:31:24.759932 31398 log.cc:1196] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:31:24.760155 31398 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Deleting consensus metadata
I20260928 06:31:24.760321 31398 master_runner.cc:315] Copying system tablet from 127.24.11.190:39467
I20260928 06:31:24.760936 31398 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.11.190:39467
I20260928 06:31:24.771745 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.761936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37442 (local address 127.24.11.190:39467)
0928 06:31:24.761988 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.761990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.765739 (+ 3749us) server_negotiation.cc:407] Connection header received
0928 06:31:24.765816 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.765817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.765830 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.765905 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.766235 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.766732 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.767508 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.767702 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.767796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.768838 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.768840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.768840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.768848 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.770580 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.770940 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.770941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.770941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.771006 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.771212 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.771213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.771213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.771335 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.771412 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.771572 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.771645 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:24.771759 31502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.761886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37442)
0928 06:31:24.765642 (+ 3756us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.765656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.765754 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.765910 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.765911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.766095 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.766151 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.766152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.766877 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.766878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.767418 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.767419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.767789 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.767789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.767853 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.768364 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.768370 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.768747 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.770672 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.770673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.770674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.770855 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.771071 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.771071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.771072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.771130 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.771404 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.771404 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.771451 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.771534 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.771647 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3731,"thread_start_us":82,"threads_started":1}
I20260928 06:31:24.772419 30767 tablet_copy_service.cc:140] P 04cb6c0b61a8482099c28c931c60bd03: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7f2757720e6d46f3a6330cb4b1bd2f6e ({username='oryx', principal='oryx/127.24.11.188@KRBTEST.COM'} at 127.0.0.1:37442)
I20260928 06:31:24.772504 30767 tablet_copy_service.cc:161] P 04cb6c0b61a8482099c28c931c60bd03: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7f2757720e6d46f3a6330cb4b1bd2f6e at {username='oryx', principal='oryx/127.24.11.188@KRBTEST.COM'} at 127.0.0.1:37442: session id = 7f2757720e6d46f3a6330cb4b1bd2f6e-00000000000000000000000000000000
I20260928 06:31:24.772986 30767 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:31:24.773569 31398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:31:24.774542 31398 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: tablet copy: Starting download of 0 data blocks...
I20260928 06:31:24.774708 31398 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: tablet copy: Starting download of 1 WAL segments...
I20260928 06:31:24.775422 31398 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:31:24.776177 30767 tablet_copy_service.cc:342] P 04cb6c0b61a8482099c28c931c60bd03: Request end of tablet copy session 7f2757720e6d46f3a6330cb4b1bd2f6e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.11.188@KRBTEST.COM'} at 127.0.0.1:37442
I20260928 06:31:24.776252 30767 tablet_copy_service.cc:433] P 04cb6c0b61a8482099c28c931c60bd03: ending tablet copy session 7f2757720e6d46f3a6330cb4b1bd2f6e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7f2757720e6d46f3a6330cb4b1bd2f6e
I20260928 06:31:24.778728 31398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.780114 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.780224 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.780120 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.780583 31398 server_base.cc:1034] running on GCE node
I20260928 06:31:24.780684 31398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.780856 31398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.781981 31398 hybrid_clock.cc:648] HybridClock initialized: now 1790577084781979 us; error 22 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.783558 31398 init.cc:377] Logged in from keytab as oryx/127.24.11.188@KRBTEST.COM (short username oryx)
I20260928 06:31:24.783844 31398 webserver.cc:460] Webserver started at http://127.24.11.188:40059/ using document root <none> and password file <none>
I20260928 06:31:24.783967 31398 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.784001 31398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.784651 31398 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:24.785218 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.785364 31398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.785410 31398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "bec1413bcdb9ed9819ec05b62470df53"
server_key_iv: "0b72aa930c42bf8c9df8f7420e3be6c3"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.785473 31398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.807330 31398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.809130 31398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.812315 31398 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.188:36087
I20260928 06:31:24.812644 31398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
I20260928 06:31:24.813447 31566 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:24.814082 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Bootstrap starting.
I20260928 06:31:24.815187 31565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.188:36087 every 8 connection(s)
I20260928 06:31:24.816412 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:24.816663 31566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: Bootstrap complete.
I20260928 06:31:24.816854 31566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } }
I20260928 06:31:24.816946 31566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f2757720e6d46f3a6330cb4b1bd2f6e, State: Initialized, Role: LEARNER
I20260928 06:31:24.816991 31566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } }
I20260928 06:31:24.817179 31566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:24.817175 31569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:24.817394 31569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: LEARNER
I20260928 06:31:24.819608 31580 catalog_manager.cc:1259] Loaded cluster ID: 658e86ed47074038a5755e21ea490458
I20260928 06:31:24.819676 31580 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: loading cluster ID for follower catalog manager: success
I20260928 06:31:24.820421 31580 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: acquiring CA information for follower catalog manager: success
I20260928 06:31:24.820717 31580 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:31:24.849396 31491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:24.849717 31491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:24.849759 31491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:24.851470 31491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:24.851539 31491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:24.851583 31491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:24.853799 31491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:36191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=36853
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:24.854084 31491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.854352 31491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.860013 31585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.860206 31491 server_base.cc:1034] running on GCE node
W20260928 06:31:24.860306 31586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.860318 31588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.860566 31491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.860821 31491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.862003 31491 hybrid_clock.cc:648] HybridClock initialized: now 1790577084861982 us; error 36 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:24.865397 31491 init.cc:377] Logged in from keytab as oryx/127.24.11.129@KRBTEST.COM (short username oryx)
I20260928 06:31:24.865880 31491 webserver.cc:460] Webserver started at http://127.24.11.129:36853/ using document root <none> and password file <none>
I20260928 06:31:24.866122 31491 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:24.866180 31491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:24.867844 31491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:24.868664 31595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.868827 31491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:24.868920 31491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "e1fb529453174b2fb2ad5e088ef7e246"
format_stamp: "Formatted at 2026-09-28 06:31:23 on dist-test-slave-38pg"
server_key: "b36334a0035e25aeb6dc1d199faf11dc"
server_key_iv: "b69ce42442d572e9aa8a6c9dd7d5c035"
server_key_version: "encryptionkey@0"
I20260928 06:31:24.869349 31491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:24.882229 31491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:24.885056 31491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:24.885303 31491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:24.885730 31491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:24.886108 31491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:24.886144 31491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.886168 31491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:24.886214 31491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:24.894251 31491 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:36191
I20260928 06:31:24.894327 31708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:36191 every 8 connection(s)
I20260928 06:31:24.894747 31491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:24.895289 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31491
I20260928 06:31:24.895448 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30943
I20260928 06:31:24.901372 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:43413
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:39467
--webserver_port=45247
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
I20260928 06:31:24.913496 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.896324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33771 (local address 127.24.11.190:39467)
0928 06:31:24.896386 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.896388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.901417 (+ 5029us) server_negotiation.cc:407] Connection header received
0928 06:31:24.902446 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.902448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.902461 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.902493 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.903832 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.904321 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.905397 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.905689 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.905750 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.909422 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.909423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.909424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.909433 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.912232 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912640 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.912641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.912641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.912697 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912933 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.912934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.912934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.913071 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.913112 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.913320 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.913404 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:31:24.913909 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.896134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:41089 (local address 127.24.11.189:34195)
0928 06:31:24.896343 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.896345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.901999 (+ 5654us) server_negotiation.cc:407] Connection header received
0928 06:31:24.902926 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.902928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.902942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.902973 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.905460 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.906193 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.908065 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.908411 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.908493 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.911010 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.911012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.911012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.911020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.912711 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.913098 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.913099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.913100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.913153 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.913365 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.913366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.913366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.913494 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.913560 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.913759 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.913835 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260928 06:31:24.913924 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.895878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:33771)
0928 06:31:24.901267 (+ 5389us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.901313 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.902319 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.902665 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.902670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.903395 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.903722 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.903725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.904418 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.904418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.905313 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.905315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.905816 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.905817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.906035 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.906827 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.906839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.909147 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.912332 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.912333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.912552 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.912772 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.912773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.912855 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.913158 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.913159 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.913202 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.913646 (+ 444us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.913750 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5376,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260928 06:31:24.914431 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.895992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:59583 (local address 127.24.11.188:36087)
0928 06:31:24.896121 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:24.896123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:24.901448 (+ 5325us) server_negotiation.cc:407] Connection header received
0928 06:31:24.902493 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:24.902495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:24.902509 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:24.902613 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:24.903826 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.904905 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.906141 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.906513 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.906593 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.909569 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:24.909570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:24.909571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:24.909579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:24.911603 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912132 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.912133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.912134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.912199 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912551 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:24.912552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:24.912552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:24.912730 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:24.912801 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:24.913814 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 06:31:24.913903 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260928 06:31:24.914428 31714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.896065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.24.11.129:41089)
0928 06:31:24.901907 (+ 5842us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.901925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.902796 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.903040 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.903041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.903466 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.903717 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.903723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.906615 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.906616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.907953 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.907956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.908504 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.908505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.908574 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.909272 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.909276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.910900 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.912789 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912789 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.912790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.913018 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.913222 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.913223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.913223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.913293 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.913609 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.913610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.913651 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.914033 (+ 382us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.914134 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5816,"mutex_wait_us":75,"thread_start_us":60,"threads_started":1}
I20260928 06:31:24.914429 31710 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:24.914563 31710 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:24.914717 31710 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:24.914902 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:24.895629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.24.11.129:59583)
0928 06:31:24.901229 (+ 5600us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:24.901332 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:24.902318 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:24.902665 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:24.902670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:24.903381 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:24.903734 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.903735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.905009 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.905010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.906036 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:24.906039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:24.906593 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:24.906594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:24.906727 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:24.907318 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:24.907324 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:24.909457 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:24.911705 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.911712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.911714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.912041 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.912274 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:24.912275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:24.912276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:24.912455 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:24.912832 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:24.912839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:24.913021 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 06:31:24.913146 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:31:24.913350 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5515,"mutex_wait_us":81,"thread_start_us":69,"threads_started":1}
I20260928 06:31:24.915349 31711 heartbeater.cc:347] Connected to a master server at 127.24.11.189:34195
I20260928 06:31:24.915376 30747 ts_manager.cc:194] Re-registered known tserver with Master: e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
I20260928 06:31:24.915406 31711 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:24.915464 31711 heartbeater.cc:511] Master 127.24.11.189:34195 requested a full tablet report, sending...
I20260928 06:31:24.915745 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:33771
I20260928 06:31:24.915791 31344 ts_manager.cc:194] Registered new tserver with Master: e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
I20260928 06:31:24.915956 31709 heartbeater.cc:347] Connected to a master server at 127.24.11.188:36087
I20260928 06:31:24.916002 31709 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:24.916059 31709 heartbeater.cc:511] Master 127.24.11.188:36087 requested a full tablet report, sending...
I20260928 06:31:24.916349 31530 ts_manager.cc:194] Registered new tserver with Master: e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
W20260928 06:31:24.987424 31717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:24.987635 31717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:24.987660 31717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:24.988801 31717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:24.988873 31717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:24.988899 31717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:24.990409 31717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:43413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=45247
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:24.990653 31717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:24.990892 31717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:24.996104 31722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:24.996106 31723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.996143 31717 server_base.cc:1034] running on GCE node
W20260928 06:31:24.996106 31725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:24.996532 31717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:24.996812 31717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:24.997984 31717 hybrid_clock.cc:648] HybridClock initialized: now 1790577084997966 us; error 35 us; skew 500 ppm
Sep 28 06:31:24 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:25.000502 31717 init.cc:377] Logged in from keytab as oryx/127.24.11.130@KRBTEST.COM (short username oryx)
I20260928 06:31:25.000993 31717 webserver.cc:460] Webserver started at http://127.24.11.130:45247/ using document root <none> and password file <none>
I20260928 06:31:25.001246 31717 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:25.001330 31717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:25.002801 31717 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:25.003527 31732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.003726 31717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:25.003815 31717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "b1f7387082dc418c8f1d39e8f51d6086"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "8bca9db05bfa657100ec94761d726daf"
server_key_iv: "9b090127c9f83e97e875e8e3ebb479b9"
server_key_version: "encryptionkey@0"
I20260928 06:31:25.004146 31717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:25.025130 31717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:25.027554 31717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:25.027670 31717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:25.027860 31717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:25.028203 31717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:25.028239 31717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.028261 31717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:25.028275 31717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.035640 31717 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:43413
I20260928 06:31:25.035676 31845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:43413 every 8 connection(s)
I20260928 06:31:25.036093 31717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:25.037175 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31717
I20260928 06:31:25.037346 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31077
I20260928 06:31:25.044463 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:39971
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:39467
--webserver_port=38639
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577084, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
I20260928 06:31:25.050973 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.039245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:60523 (local address 127.24.11.188:36087)
0928 06:31:25.039310 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.039312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.039320 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:25.039344 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.039345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.039358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.039417 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.040112 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.040601 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.042383 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.042721 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.042802 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.046730 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.046732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.046732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.046742 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.048684 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.049868 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.049869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.049870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.049937 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.050203 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.050204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.050205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.050339 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.050421 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.050735 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.050840 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:25.051817 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.037234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.24.11.130:60523)
0928 06:31:25.038216 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.038228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.038778 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.039443 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.039444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.039765 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.040026 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.040028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.040825 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.040826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.042285 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.042288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.042843 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.042844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.043003 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.043735 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.043747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.046610 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.048827 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.048834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.048837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.049778 (+ 941us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.050004 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.050005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.050005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.050128 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.050413 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.050419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.050567 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.051344 (+ 777us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.051545 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":80,"thread_start_us":90,"threads_started":1}
I20260928 06:31:25.051962 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.036857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:44521 (local address 127.24.11.190:39467)
0928 06:31:25.036938 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.036940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.037232 (+ 292us) server_negotiation.cc:407] Connection header received
0928 06:31:25.038869 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.038871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.038883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.038940 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.039879 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.040729 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.042865 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.043221 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.043294 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.046712 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.046714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.046715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.046723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.049474 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.051012 (+ 1538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.051013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.051013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.051080 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.051302 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.051302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.051303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.051446 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.051506 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.051719 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.051811 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:25.052413 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.036878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:44521)
0928 06:31:25.037097 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.037134 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.038757 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.039014 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.039018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.039387 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.039756 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.039762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.040838 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.040838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.042285 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.042288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.043302 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.043303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.043429 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.044057 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.044061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.046602 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.050673 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.050674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.050674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.050901 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.051158 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.051159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.051159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.051223 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.051542 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.051543 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.051589 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.052054 (+ 465us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.052172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":41,"thread_start_us":132,"threads_started":1}
I20260928 06:31:25.052889 31846 heartbeater.cc:347] Connected to a master server at 127.24.11.188:36087
I20260928 06:31:25.053016 31846 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.053231 31846 heartbeater.cc:511] Master 127.24.11.188:36087 requested a full tablet report, sending...
I20260928 06:31:25.053735 31530 ts_manager.cc:194] Registered new tserver with Master: b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413)
I20260928 06:31:25.054059 31847 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:25.054111 31847 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.054181 31847 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:25.054188 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.037410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.24.11.130:48125)
0928 06:31:25.044218 (+ 6808us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.044230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.044320 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.044864 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.044865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.045030 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.045120 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.045121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.046088 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.046089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.046941 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.046943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.048411 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.048412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.048486 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.048920 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.048925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.050568 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.052514 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.052515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.052515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.052741 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.053055 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.053056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.053056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.053115 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.053423 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.053424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.053464 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.053560 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.053660 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6789,"thread_start_us":148,"threads_started":1}
I20260928 06:31:25.054437 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.038301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:48125 (local address 127.24.11.189:34195)
0928 06:31:25.038382 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.038384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.044291 (+ 5907us) server_negotiation.cc:407] Connection header received
0928 06:31:25.044395 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.044419 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.044433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.044478 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.045216 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.046001 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.047607 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.048311 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.048408 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.050664 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.050666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.050666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.050674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.052421 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.052831 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.052832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.052833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.052955 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.053205 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.053206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.053206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.053359 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.053418 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.054248 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.054344 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:31:25.054450 30747 ts_manager.cc:194] Re-registered known tserver with Master: b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413)
I20260928 06:31:25.054827 31848 heartbeater.cc:347] Connected to a master server at 127.24.11.189:34195
I20260928 06:31:25.054893 31848 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.054906 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:44521
I20260928 06:31:25.054965 31848 heartbeater.cc:511] Master 127.24.11.189:34195 requested a full tablet report, sending...
I20260928 06:31:25.055250 31344 ts_manager.cc:194] Registered new tserver with Master: b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413)
W20260928 06:31:25.130231 31853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:25.130434 31853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:25.130456 31853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:25.131639 31853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:25.131681 31853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:25.131707 31853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:25.133168 31853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=38639
--tserver_master_addrs=127.24.11.190:39467,127.24.11.189:34195,127.24.11.188:36087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:25.133421 31853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:25.133682 31853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:25.139798 31858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:25.139827 31853 server_base.cc:1034] running on GCE node
W20260928 06:31:25.139847 31859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:25.139806 31861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:25.140136 31853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:25.140383 31853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:25.141587 31853 hybrid_clock.cc:648] HybridClock initialized: now 1790577085141563 us; error 29 us; skew 500 ppm
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577085, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:25.144222 31853 init.cc:377] Logged in from keytab as oryx/127.24.11.131@KRBTEST.COM (short username oryx)
I20260928 06:31:25.144693 31853 webserver.cc:460] Webserver started at http://127.24.11.131:38639/ using document root <none> and password file <none>
I20260928 06:31:25.144936 31853 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:25.145025 31853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:25.146628 31853 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:25.147228 31868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.147413 31853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:25.147514 31853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
format_stamp: "Formatted at 2026-09-28 06:31:24 on dist-test-slave-38pg"
server_key: "ee6ab8b7e6ef7fc8c8c48855e55ed0c0"
server_key_iv: "fcf12d1899ac54312656722e3eb11b8e"
server_key_version: "encryptionkey@0"
I20260928 06:31:25.147861 31853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:25.158779 31853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:25.160717 31853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:25.160880 31853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:25.161120 31853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:25.161427 31853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:25.161480 31853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.161522 31853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:25.161572 31853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:25.167686 31853 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:39971
I20260928 06:31:25.167726 31981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:39971 every 8 connection(s)
I20260928 06:31:25.168147 31853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:25.170076 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 31853
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577085, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577085, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.189@KRBTEST.COM
Sep 28 06:31:25 dist-test-slave-38pg krb5kdc[30702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577085, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.188@KRBTEST.COM
I20260928 06:31:25.182219 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.169296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:58901 (local address 127.24.11.190:39467)
0928 06:31:25.169340 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.169342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.169406 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:31:25.170087 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.170089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.170103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.170157 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.171644 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.172414 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174424 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174632 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174664 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.178699 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.178701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.178701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.178709 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.180634 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.181104 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.181105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.181105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.181164 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.181410 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.181411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.181411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.181569 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.181628 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.182013 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.182100 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:31:25.182964 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.169468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:33579 (local address 127.24.11.189:34195)
0928 06:31:25.169512 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.169514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.169654 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:31:25.170091 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.170093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.170105 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.170138 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.171644 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.172420 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174398 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174687 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174772 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.178736 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.178737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.178738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.178744 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.180981 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.182121 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.182122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.182123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.182186 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.182405 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.182406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.182406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.182530 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.182573 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.182762 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.182839 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:25.183075 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.171558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.24.11.189:34195)
0928 06:31:25.171655 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.171657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.172028 (+ 371us) server_negotiation.cc:407] Connection header received
0928 06:31:25.172128 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.172129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.172142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.172181 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.173264 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174037 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.175948 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.176232 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.176268 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.177684 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.177685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.177686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.177692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.180159 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.180607 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.180608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.180608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.180669 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.181311 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.181312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.181313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.182480 (+ 1167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.182575 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.182847 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.182909 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260928 06:31:25.183429 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.169407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.24.11.131:33579)
0928 06:31:25.169596 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.169606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.169994 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.170329 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.170332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.171181 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.171526 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.171532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.172653 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.172653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.174321 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.174323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174776 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.174980 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.175771 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.175782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.178647 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.181358 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.181359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.181359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.181712 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.182262 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.182263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.182264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.182328 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.182607 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.182608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.182649 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.183054 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.183158 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":388,"thread_start_us":67,"threads_started":1}
I20260928 06:31:25.184031 31984 heartbeater.cc:347] Connected to a master server at 127.24.11.189:34195
I20260928 06:31:25.184113 31984 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.184252 31984 heartbeater.cc:511] Master 127.24.11.189:34195 requested a full tablet report, sending...
I20260928 06:31:25.184820 31715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.168918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40589 (local address 127.24.11.188:36087)
0928 06:31:25.168967 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.168970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.169230 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:31:25.170091 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.170093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.170105 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.170138 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.174845 (+ 4707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.176024 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.177323 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.177683 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.177756 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.180123 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.180124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.180124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.180133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.182339 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.183460 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.183461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.183461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.183539 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.183797 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.183798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.183798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.183949 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.183992 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.184626 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.184719 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:25.185046 31343 ts_manager.cc:194] Registered new tserver with Master: 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:25.185379 31987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.169229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:58901)
0928 06:31:25.169340 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.169350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.169968 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.170624 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.170625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.171057 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.171525 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.171532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.172644 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.172645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.174321 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.174323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174771 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.174772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.174980 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.175824 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.175829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.178583 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.180712 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.180718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.180721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.181017 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.181225 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.181226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.181226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.181341 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.181664 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.181670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.181851 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.182637 (+ 786us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.182829 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":52,"thread_start_us":59,"threads_started":1}
I20260928 06:31:25.185765 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.168920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.24.11.131:40589)
0928 06:31:25.169097 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.169133 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.169956 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.170328 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.170332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.173017 (+ 2685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:25.173351 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.173353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.176108 (+ 2755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.176109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.177230 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.177233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.177813 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.177814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.177941 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.178436 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:25.178441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:25.180018 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:25.183132 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.183133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.183134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.183320 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.183659 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:25.183660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:25.183660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:25.183720 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:25.184455 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:25.184455 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:25.184505 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.185160 (+ 655us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.185266 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":80,"thread_start_us":95,"threads_started":1}
I20260928 06:31:25.186085 31983 heartbeater.cc:347] Connected to a master server at 127.24.11.190:39467
I20260928 06:31:25.186095 31982 heartbeater.cc:347] Connected to a master server at 127.24.11.188:36087
I20260928 06:31:25.186146 31983 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.186177 31982 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:25.186236 31983 heartbeater.cc:511] Master 127.24.11.190:39467 requested a full tablet report, sending...
I20260928 06:31:25.186271 31982 heartbeater.cc:511] Master 127.24.11.188:36087 requested a full tablet report, sending...
I20260928 06:31:25.186507 30747 ts_manager.cc:194] Re-registered known tserver with Master: 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:25.186538 31530 ts_manager.cc:194] Registered new tserver with Master: 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:25.186975 30747 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:58901
I20260928 06:31:25.187676 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.171786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.24.11.188:36087)
0928 06:31:25.172640 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.172642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.172650 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:25.172674 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.172674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.172686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.173363 (+ 677us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.173400 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.174125 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.176571 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.177765 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.181940 (+ 4175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.183268 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.183269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.183269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.183276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.186247 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.186730 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.186731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.186732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.186789 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.187039 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.187040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.187040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.187206 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.187338 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.187474 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.187557 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":59,"threads_started":1}
I20260928 06:31:25.190164 31994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.171229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.24.11.190:39467)
0928 06:31:25.171357 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.171359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.171993 (+ 634us) server_negotiation.cc:407] Connection header received
0928 06:31:25.172102 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.172104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.172116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.172164 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.180635 (+ 8471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.181395 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.184542 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.184832 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.184877 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.187661 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.187662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.187663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.187669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.189153 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.189480 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.189481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.189481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.189576 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.189738 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.189739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.189739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.189846 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.189918 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.190008 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.190066 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260928 06:31:25.191484 30747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:25.193050 30747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:25.200721 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:39971 (local address 127.0.0.1:51438)
0928 06:31:25.197304 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.197315 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.197427 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.197694 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.197695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.197697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.197752 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.197754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.198940 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.198941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.199813 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.199815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.200479 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.200480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.200538 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.200542 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.200600 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.200642 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260928 06:31:25.200730 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.24.11.131:39971)
0928 06:31:25.197495 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.197499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.197510 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:25.197578 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.197581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.197598 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.197730 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.197864 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.198842 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.199893 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.200395 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.200481 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.200560 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.200615 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.200653 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":109,"threads_started":1}
I20260928 06:31:25.201617 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:43413 (local address 127.0.0.1:34164)
0928 06:31:25.197210 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.197223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.197270 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.198031 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.198032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.198033 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.198056 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.198056 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.199315 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.199316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.200437 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.200439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.201373 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.201374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.201434 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.201437 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.201496 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.201556 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":4}
I20260928 06:31:25.201880 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.24.11.130:43413)
0928 06:31:25.197868 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.197874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.197883 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:25.197924 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.197926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.197941 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.198021 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.198131 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.199216 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.200534 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.201250 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.201358 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.201443 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.201506 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.201577 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":81,"threads_started":1}
I20260928 06:31:25.202239 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.24.11.129:36191)
0928 06:31:25.198137 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.198141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.198275 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:31:25.198321 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.198323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.198339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.198423 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.198545 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.199540 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.200764 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.201453 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.201496 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.201883 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.202123 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.202167 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":83,"threads_started":1}
I20260928 06:31:25.202442 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.197205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36191 (local address 127.0.0.1:44054)
0928 06:31:25.198194 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.198203 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.198259 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.198427 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.198428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.198430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.198486 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.198488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.199623 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.199623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.200674 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.200675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.201967 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.201968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.202026 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.202030 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.202333 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.202382 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260928 06:31:25.202394 31916 tablet_service.cc:1467] Processing CreateTablet for tablet faf4bcb2976c42b397585df970154dfe (DEFAULT_TABLE table=test-table [id=2c5dd46ab2324bdcbdb5cd38ba716dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:25.202518 31780 tablet_service.cc:1467] Processing CreateTablet for tablet faf4bcb2976c42b397585df970154dfe (DEFAULT_TABLE table=test-table [id=2c5dd46ab2324bdcbdb5cd38ba716dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:25.202791 31780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf4bcb2976c42b397585df970154dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:25.202791 31916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf4bcb2976c42b397585df970154dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:25.206220 31643 tablet_service.cc:1467] Processing CreateTablet for tablet faf4bcb2976c42b397585df970154dfe (DEFAULT_TABLE table=test-table [id=2c5dd46ab2324bdcbdb5cd38ba716dc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:25.206403 32008 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Bootstrap starting.
I20260928 06:31:25.206568 31643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf4bcb2976c42b397585df970154dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:25.207206 32008 tablet_bootstrap.cc:654] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:25.207530 32008 log.cc:824] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:25.207866 32009 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Bootstrap starting.
I20260928 06:31:25.208329 32008 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: No bootstrap required, opened a new log
I20260928 06:31:25.208377 32008 ts_tablet_manager.cc:1397] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:25.208534 32009 tablet_bootstrap.cc:654] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:25.208839 32009 log.cc:824] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:25.209649 32009 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: No bootstrap required, opened a new log
I20260928 06:31:25.209715 32009 ts_tablet_manager.cc:1397] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:25.210423 32012 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Bootstrap starting.
I20260928 06:31:25.210500 32008 raft_consensus.cc:348] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.210660 32008 raft_consensus.cc:374] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:25.210704 32008 raft_consensus.cc:729] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbb1b5f8d2a43b0b8fb98a277f8edce, State: Initialized, Role: FOLLOWER
I20260928 06:31:25.210860 32008 consensus_queue.cc:260] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.211119 32008 ts_tablet_manager.cc:1428] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:25.211114 32012 tablet_bootstrap.cc:654] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:25.211447 32012 log.cc:824] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:25.211500 31983 heartbeater.cc:503] Master 127.24.11.190:39467 was elected leader, sending a full tablet report...
I20260928 06:31:25.213003 32009 raft_consensus.cc:348] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.213156 32009 raft_consensus.cc:374] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:25.213200 32009 raft_consensus.cc:729] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f7387082dc418c8f1d39e8f51d6086, State: Initialized, Role: FOLLOWER
I20260928 06:31:25.213307 32009 consensus_queue.cc:260] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.213613 32009 ts_tablet_manager.cc:1428] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 06:31:25.213646 31847 heartbeater.cc:503] Master 127.24.11.190:39467 was elected leader, sending a full tablet report...
I20260928 06:31:25.214421 32012 tablet_bootstrap.cc:492] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: No bootstrap required, opened a new log
I20260928 06:31:25.214488 32012 ts_tablet_manager.cc:1397] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:31:25.215945 32012 raft_consensus.cc:348] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.216063 32012 raft_consensus.cc:374] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:25.216084 32012 raft_consensus.cc:729] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fb529453174b2fb2ad5e088ef7e246, State: Initialized, Role: FOLLOWER
I20260928 06:31:25.216214 32012 consensus_queue.cc:260] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.216454 32012 ts_tablet_manager.cc:1428] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:25.216553 31710 heartbeater.cc:503] Master 127.24.11.190:39467 was elected leader, sending a full tablet report...
W20260928 06:31:25.286969 31849 tablet.cc:2369] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:25.349999 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.347490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.24.11.188:36087)
0928 06:31:25.347561 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.347563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.347573 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:25.347603 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.347604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.347622 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.347685 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.347808 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.348297 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.349208 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.349696 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.349777 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.349840 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.349903 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.349933 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:31:25.350078 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.347405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46386)
0928 06:31:25.347491 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.347503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.347556 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.347700 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.347700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.347702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.347750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.347752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.348379 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.348380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.349096 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.349098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.349779 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.349780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.349827 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.349831 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.349897 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.349948 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 06:31:25.350508 31540 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 06:31:25.351550 32017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04cb6c0b61a8482099c28c931c60bd03. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:25.351667 32017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: LEARNER
W20260928 06:31:25.395870 31712 tablet.cc:2369] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:25.409964 32016 raft_consensus.cc:482] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:25.410089 32016 raft_consensus.cc:504] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.410506 32016 leader_election.cc:288] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413), 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:25.414503 31713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.410723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:39971 (local address 127.24.11.129:46627)
0928 06:31:25.410781 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.410795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.410846 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.411113 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.411115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.411118 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.411198 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.411201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.412137 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.412138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.413394 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.413396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.414203 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.414204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.414291 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.414296 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.414360 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.414400 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":10}
I20260928 06:31:25.414544 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.410830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:46627 (local address 127.24.11.131:39971)
0928 06:31:25.410899 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.410901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.410912 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:25.410945 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.410947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.410965 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.411040 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.411276 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.412056 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.413499 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.414097 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.414182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.414191 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.414410 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.414466 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:31:25.414515 31716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.411029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:43413 (local address 127.24.11.129:54297)
0928 06:31:25.411105 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:25.411118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:25.411180 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:25.411331 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:25.411336 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:25.411338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:25.411375 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.411376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.412236 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.412236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.413394 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:25.413396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.414203 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.414204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:25.414298 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.414300 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:25.414360 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:31:25.414400 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":35}
I20260928 06:31:25.414657 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.410874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:54297 (local address 127.24.11.130:43413)
0928 06:31:25.410944 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.410945 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.411185 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:31:25.411251 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.411252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.411262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.411320 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:25.411438 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.412166 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.413499 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.414123 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.414189 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.414197 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.414537 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.414587 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:25.415009 31936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf4bcb2976c42b397585df970154dfe" candidate_uuid: "e1fb529453174b2fb2ad5e088ef7e246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" is_pre_election: true
I20260928 06:31:25.415064 31800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf4bcb2976c42b397585df970154dfe" candidate_uuid: "e1fb529453174b2fb2ad5e088ef7e246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f7387082dc418c8f1d39e8f51d6086" is_pre_election: true
I20260928 06:31:25.415179 31936 raft_consensus.cc:2454] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fb529453174b2fb2ad5e088ef7e246 in term 0.
I20260928 06:31:25.415210 31800 raft_consensus.cc:2454] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fb529453174b2fb2ad5e088ef7e246 in term 0.
I20260928 06:31:25.415445 31597 leader_election.cc:302] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dbb1b5f8d2a43b0b8fb98a277f8edce, e1fb529453174b2fb2ad5e088ef7e246; no voters:
I20260928 06:31:25.415622 32016 raft_consensus.cc:2781] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:25.415705 32016 raft_consensus.cc:482] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:25.415740 32016 raft_consensus.cc:3037] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:25.416483 32016 raft_consensus.cc:504] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
I20260928 06:31:25.416638 32016 leader_election.cc:288] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [CANDIDATE]: Term 1 election: Requested vote from peers b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413), 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971)
I20260928 06:31:25.416810 31936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf4bcb2976c42b397585df970154dfe" candidate_uuid: "e1fb529453174b2fb2ad5e088ef7e246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
I20260928 06:31:25.416854 31800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf4bcb2976c42b397585df970154dfe" candidate_uuid: "e1fb529453174b2fb2ad5e088ef7e246" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1f7387082dc418c8f1d39e8f51d6086"
I20260928 06:31:25.416905 31936 raft_consensus.cc:3037] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:25.416922 31800 raft_consensus.cc:3037] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:25.417868 31800 raft_consensus.cc:2454] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fb529453174b2fb2ad5e088ef7e246 in term 1.
I20260928 06:31:25.417865 31936 raft_consensus.cc:2454] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fb529453174b2fb2ad5e088ef7e246 in term 1.
I20260928 06:31:25.418046 31596 leader_election.cc:302] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f7387082dc418c8f1d39e8f51d6086, e1fb529453174b2fb2ad5e088ef7e246; no voters:
I20260928 06:31:25.418143 32016 raft_consensus.cc:2781] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:25.418254 32016 raft_consensus.cc:686] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 1 LEADER]: Becoming Leader. State: Replica: e1fb529453174b2fb2ad5e088ef7e246, State: Running, Role: LEADER
I20260928 06:31:25.418360 32016 consensus_queue.cc:237] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } }
W20260928 06:31:25.419166 31985 tablet.cc:2369] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:25.419253 30747 catalog_manager.cc:5495] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129). New cstate: current_term: 1 leader_uuid: "e1fb529453174b2fb2ad5e088ef7e246" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:25.419950 31305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: NON_VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: true } } }
I20260928 06:31:25.420042 31305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:25.420830 31494 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: attempting to promote NON_VOTER 7f2757720e6d46f3a6330cb4b1bd2f6e to VOTER
I20260928 06:31:25.420954 31494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } }
I20260928 06:31:25.421474 31354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Refusing update from remote peer 04cb6c0b61a8482099c28c931c60bd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:31:25.421558 31540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 LEARNER]: Refusing update from remote peer 04cb6c0b61a8482099c28c931c60bd03: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 06:31:25.421742 31305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:31:25.421844 31494 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 06:31:25.422396 31494 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.423207 32023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.423306 32023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:25.423389 32023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.423467 32023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:25.423213 31494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.423521 31494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:25.423596 31354 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.423676 31540 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.424642 32027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.424737 32027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:25.424777 32017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04cb6c0b61a8482099c28c931c60bd03" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:25.424865 32017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:25.427522 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.425337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.24.11.129:36191)
0928 06:31:25.425381 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.425383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.425417 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:31:25.425484 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.425484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.425501 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.425619 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:25.425709 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.426169 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.426898 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.427096 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.427145 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.427271 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:25.427363 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:25.427424 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.427456 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.427472 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:25.429316 31936 raft_consensus.cc:1261] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Refusing update from remote peer e1fb529453174b2fb2ad5e088ef7e246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:25.429307 31800 raft_consensus.cc:1261] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Refusing update from remote peer e1fb529453174b2fb2ad5e088ef7e246: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:25.429688 32021 consensus_queue.cc:1035] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:25.429885 32016 consensus_queue.cc:1035] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:25.433497 32030 mvcc.cc:205] Tried to move back new op lower bound from 7334203741915639808 to 7334203741874552832. Current Snapshot: MvccSnapshot[applied={T|T < 7334203741915639808}]
I20260928 06:31:25.433975 32028 mvcc.cc:205] Tried to move back new op lower bound from 7334203741915639808 to 7334203741874552832. Current Snapshot: MvccSnapshot[applied={T|T < 7334203741915639808}]
I20260928 06:31:25.436554 30747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:25.436632 30747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:25.438151 30747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04cb6c0b61a8482099c28c931c60bd03: Sending DeleteTablet for 3 replicas of tablet faf4bcb2976c42b397585df970154dfe
I20260928 06:31:25.438510 31916 tablet_service.cc:1514] Processing DeleteTablet for tablet faf4bcb2976c42b397585df970154dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:25 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:51438
I20260928 06:31:25.439317 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 30715
I20260928 06:31:25.439767 31780 tablet_service.cc:1514] Processing DeleteTablet for tablet faf4bcb2976c42b397585df970154dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:25 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:34164
I20260928 06:31:25.440034 31643 tablet_service.cc:1514] Processing DeleteTablet for tablet faf4bcb2976c42b397585df970154dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:25 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:44054
I20260928 06:31:25.440232 32040 tablet_replica.cc:330] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: stopping tablet replica
I20260928 06:31:25.440346 32040 raft_consensus.cc:2229] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:25.440481 32040 raft_consensus.cc:2258] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:25.440966 32040 ts_tablet_manager.cc:1905] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:25.442974 32040 ts_tablet_manager.cc:1918] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:25.443050 32040 log.cc:1196] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/faf4bcb2976c42b397585df970154dfe
I20260928 06:31:25.443311 32040 ts_tablet_manager.cc:1939] T faf4bcb2976c42b397585df970154dfe P e1fb529453174b2fb2ad5e088ef7e246: Deleting consensus metadata
W20260928 06:31:25.445351 31598 connection.cc:535] server connection from 127.0.0.1:44054 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:31:25.446508 32039 tablet_replica.cc:330] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: stopping tablet replica
I20260928 06:31:25.446624 32039 raft_consensus.cc:2229] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:25.446732 32039 raft_consensus.cc:2258] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:25.446898 32037 tablet_replica.cc:330] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: stopping tablet replica
I20260928 06:31:25.446992 32037 raft_consensus.cc:2229] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:25.447093 32037 raft_consensus.cc:2258] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:25.447538 32037 ts_tablet_manager.cc:1905] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:25.447539 32039 ts_tablet_manager.cc:1905] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:25.449425 32039 ts_tablet_manager.cc:1918] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:25.449505 32039 log.cc:1196] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/faf4bcb2976c42b397585df970154dfe
I20260928 06:31:25.449872 32039 ts_tablet_manager.cc:1939] T faf4bcb2976c42b397585df970154dfe P b1f7387082dc418c8f1d39e8f51d6086: Deleting consensus metadata
W20260928 06:31:25.450250 31736 connection.cc:441] server connection from 127.0.0.1:34164 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34164 (request call id 1) could send its response
I20260928 06:31:25.450472 32037 ts_tablet_manager.cc:1918] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:25.450553 32037 log.cc:1196] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/faf4bcb2976c42b397585df970154dfe
I20260928 06:31:25.450881 32037 ts_tablet_manager.cc:1939] T faf4bcb2976c42b397585df970154dfe P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting consensus metadata
I20260928 06:31:25.450934 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.438871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.24.11.188:36087)
0928 06:31:25.438935 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.438938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.439061 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:31:25.439158 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.439158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.439168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.439197 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.441117 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.441897 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.442971 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.443272 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.443314 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.445165 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.445167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.445168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.445176 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.446994 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.449624 (+ 2630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.449626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.449626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.449696 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.449985 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.449986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.449987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.450152 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.450242 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.450704 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.450825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 06:31:25.451234 31872 connection.cc:441] server connection from 127.0.0.1:51438 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51438 (request call id 1) could send its response
I20260928 06:31:25.461051 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.453982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.24.11.189:34195)
0928 06:31:25.454032 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.454034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.454043 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:25.454068 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.454069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.454082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.454135 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.454832 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.455491 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.456252 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.456504 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.456579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.458123 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.458124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.458125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.458133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.459789 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.460190 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.460191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.460191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.460252 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.460469 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.460470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.460470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.460614 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.460715 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.460833 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.460928 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:31:25.461971 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:25.454101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.24.11.188:36087)
0928 06:31:25.454144 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:25.454146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:25.454259 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:31:25.454289 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:25.454289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:25.454299 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:25.454349 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:25.455229 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.455974 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.457060 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:25.457327 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:25.457407 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:25.458775 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:25.458776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:25.458777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:25.458784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:25.460609 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.461023 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.461024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.461025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.461078 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:25.461291 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:25.461292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:25.461292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:25.461431 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:25.461502 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:25.461788 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:31:25.461866 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:26.434652 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.434208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:59327)
0928 06:31:26.434507 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.434596 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":86,"threads_started":1}
I20260928 06:31:26.434660 32048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.434208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:41669)
0928 06:31:26.434528 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.434607 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":66,"threads_started":1}
W20260928 06:31:26.434886 31847 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:39467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
W20260928 06:31:26.435213 31983 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:39467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.435266 32049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.435019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:46985)
0928 06:31:26.435144 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.435223 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
W20260928 06:31:26.435434 31710 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:39467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.938894 32050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04cb6c0b61a8482099c28c931c60bd03)
I20260928 06:31:26.938894 32051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 04cb6c0b61a8482099c28c931c60bd03)
I20260928 06:31:26.939052 32051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } }
I20260928 06:31:26.939024 32050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } }
I20260928 06:31:26.939332 32051 leader_election.cc:288] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467), 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188:36087)
I20260928 06:31:26.939347 32050 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467), 3b545cd2ded54018a84389c63c1e4103 (127.24.11.189:34195)
I20260928 06:31:26.939744 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37940)
0928 06:31:26.939626 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.939700 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
I20260928 06:31:26.939882 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37936)
0928 06:31:26.939777 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.939837 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":66,"threads_started":1}
W20260928 06:31:26.940006 31515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
W20260928 06:31:26.940282 31329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.940454 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.940373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37942)
0928 06:31:26.940408 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.940431 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:31:26.940541 31515 leader_election.cc:334] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467): Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.940806 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.940688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37944)
0928 06:31:26.940750 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.940782 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:31:26.941275 31329 leader_election.cc:334] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467): Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.943500 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:36087 (local address 127.0.0.1:46406)
0928 06:31:26.939778 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.939789 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:26.939846 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:26.940207 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:26.940209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:26.940211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:26.940271 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:26.940274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.941231 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.941232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:26.942479 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:26.942482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.943270 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.943271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:26.943318 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:26.943322 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:26.943368 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:31:26.943401 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":34,"threads_started":1}
I20260928 06:31:26.943521 32057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.24.11.188:36087)
0928 06:31:26.940059 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:26.940061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:26.940069 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:26.940096 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:26.940097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:26.940107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:26.940188 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:26.940362 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.941151 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.942584 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.943177 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.943264 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:26.943276 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:26.943396 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 06:31:26.943444 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260928 06:31:26.943769 32056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.24.11.189:34195)
0928 06:31:26.939933 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:26.939935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:26.940112 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:31:26.940522 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:26.940523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:26.940534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:26.940577 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:26.940777 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.941466 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.942762 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.943388 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.943454 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:26.943464 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:26.943665 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:31:26.943714 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":49,"threads_started":1}
I20260928 06:31:26.943778 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.939680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:34195 (local address 127.0.0.1:60062)
0928 06:31:26.940000 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.940012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:26.940148 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:26.940643 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:26.940644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:26.940646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:26.940697 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:26.940702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.941563 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.941564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:26.942675 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:26.942678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:26.943473 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:26.943474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:26.943539 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:26.943543 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:26.943656 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:31:26.943712 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":50,"threads_started":1}
I20260928 06:31:26.943861 31540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b545cd2ded54018a84389c63c1e4103" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" is_pre_election: true
I20260928 06:31:26.943992 31540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b545cd2ded54018a84389c63c1e4103 in term 1.
I20260928 06:31:26.944159 31329 leader_election.cc:302] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3b545cd2ded54018a84389c63c1e4103, 7f2757720e6d46f3a6330cb4b1bd2f6e; no voters: 04cb6c0b61a8482099c28c931c60bd03
I20260928 06:31:26.944265 32051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:31:26.944346 32051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 04cb6c0b61a8482099c28c931c60bd03)
I20260928 06:31:26.944244 31354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3b545cd2ded54018a84389c63c1e4103" is_pre_election: true
I20260928 06:31:26.944384 32051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:26.945127 32051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } }
I20260928 06:31:26.945283 32051 leader_election.cc:288] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 election: Requested vote from peers 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467), 7f2757720e6d46f3a6330cb4b1bd2f6e (127.24.11.188:36087)
I20260928 06:31:26.945314 31354 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f2757720e6d46f3a6330cb4b1bd2f6e in current term 2: Already voted for candidate 3b545cd2ded54018a84389c63c1e4103 in this term.
I20260928 06:31:26.945482 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.945379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37946)
0928 06:31:26.945417 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.945458 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:31:26.945501 31540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b545cd2ded54018a84389c63c1e4103" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e"
I20260928 06:31:26.945623 31540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:26.945732 31515 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f2757720e6d46f3a6330cb4b1bd2f6e; no voters: 04cb6c0b61a8482099c28c931c60bd03, 3b545cd2ded54018a84389c63c1e4103
I20260928 06:31:26.946452 31540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b545cd2ded54018a84389c63c1e4103 in term 2.
I20260928 06:31:26.946565 32050 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:31:26.946620 31329 leader_election.cc:302] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b545cd2ded54018a84389c63c1e4103, 7f2757720e6d46f3a6330cb4b1bd2f6e; no voters:
I20260928 06:31:26.946699 32051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:26.946851 32051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [term 2 LEADER]: Becoming Leader. State: Replica: 3b545cd2ded54018a84389c63c1e4103, State: Running, Role: LEADER
I20260928 06:31:26.946925 32051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } }
I20260928 06:31:26.947014 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:26.946915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:37954)
0928 06:31:26.946961 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:26.946989 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:31:26.947077 31329 leader_election.cc:334] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467): Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
I20260928 06:31:26.947417 32060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b545cd2ded54018a84389c63c1e4103. Latest consensus state: current_term: 2 leader_uuid: "3b545cd2ded54018a84389c63c1e4103" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:26.947518 32060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:26.947714 32062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:26.948109 32062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:26.948205 32062 catalog_manager.cc:1259] Loaded cluster ID: 658e86ed47074038a5755e21ea490458
I20260928 06:31:26.948261 32062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:26.948436 32062 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:26.948531 32062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Loaded TSK: 0
I20260928 06:31:26.948695 32062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:27.003152 31343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:27.004788 31343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:27.006186 31540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [term 2 FOLLOWER]: Refusing update from remote peer 3b545cd2ded54018a84389c63c1e4103: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:31:27.006220 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.006097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:38002)
0928 06:31:27.006151 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.006184 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:31:27.006551 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.006470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.0.0.1:38018)
0928 06:31:27.006503 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.006527 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 06:31:27.006614 32060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 06:31:27.006767 31329 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 -> Peer 04cb6c0b61a8482099c28c931c60bd03 (127.24.11.190:39467): Couldn't send request to peer 04cb6c0b61a8482099c28c931c60bd03. Status: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:31:27.007336 32050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b545cd2ded54018a84389c63c1e4103. Latest consensus state: current_term: 2 leader_uuid: "3b545cd2ded54018a84389c63c1e4103" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:27.007468 32050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:27.007838 32063 mvcc.cc:205] Tried to move back new op lower bound from 7334203748374261760 to 7334203748135677952. Current Snapshot: MvccSnapshot[applied={T|T < 7334203748374261760}]
I20260928 06:31:27.007936 32051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3b545cd2ded54018a84389c63c1e4103" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:27.008018 32051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:27.008213 32050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3b545cd2ded54018a84389c63c1e4103" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "04cb6c0b61a8482099c28c931c60bd03" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39467 } } peers { permanent_uuid: "3b545cd2ded54018a84389c63c1e4103" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 34195 } attrs { promote: false } } peers { permanent_uuid: "7f2757720e6d46f3a6330cb4b1bd2f6e" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 36087 } attrs { promote: false } } }
I20260928 06:31:27.008303 32050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2757720e6d46f3a6330cb4b1bd2f6e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:31:27.013175 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.010072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.24.11.131:39971)
0928 06:31:27.010330 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.010333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.010340 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:27.010371 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.010371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.010379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.010457 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:27.010574 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011081 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.012320 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.012839 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.012940 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.012953 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.013069 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.013107 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
I20260928 06:31:27.013173 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.009832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:39971 (local address 127.0.0.1:51446)
0928 06:31:27.009887 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.009897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.009934 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.010459 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:27.010460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:27.010461 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:27.010520 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.010521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011165 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.011167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.012221 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.012223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.012940 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.012941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.012998 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.013002 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:27.013055 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:31:27.013099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:31:27.013339 32068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.009929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:43413 (local address 127.0.0.1:34174)
0928 06:31:27.010090 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.010100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.010217 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.010622 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:27.010622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:27.010624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:27.010656 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.010657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011528 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.011530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.012389 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.012391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.013107 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.013108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.013166 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.013169 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:27.013216 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:31:27.013264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260928 06:31:27.013594 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.009842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36191 (local address 127.0.0.1:44078)
0928 06:31:27.009908 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.009919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.009982 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.010476 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:27.010477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:27.010480 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:27.010540 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.010542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011481 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.011482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.012575 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.012577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.013296 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.013296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.013350 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.013353 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:27.013396 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:31:27.013481 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":25}
I20260928 06:31:27.013640 31916 tablet_service.cc:1467] Processing CreateTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 (DEFAULT_TABLE table=test-table [id=81f0b0e12f054a59a1b976d5c5c06359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:27.013792 31916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fce7addb0e943aaaa837b8e6eab7f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:27.013960 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.010215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.24.11.129:36191)
0928 06:31:27.010325 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.010327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.010335 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:27.010361 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.010362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.010380 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.010606 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:27.010645 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011393 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.012658 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.013211 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.013791 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.013807 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.013826 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.013870 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260928 06:31:27.014179 31643 tablet_service.cc:1467] Processing CreateTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 (DEFAULT_TABLE table=test-table [id=81f0b0e12f054a59a1b976d5c5c06359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:27.014323 31643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fce7addb0e943aaaa837b8e6eab7f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:27.015348 32072 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Bootstrap starting.
I20260928 06:31:27.015530 32073 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: Bootstrap starting.
I20260928 06:31:27.015820 32072 tablet_bootstrap.cc:654] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:27.015906 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.010083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34174 (local address 127.24.11.130:43413)
0928 06:31:27.010198 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.010200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.010211 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:27.010501 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.010506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.010517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.010578 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:27.010726 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.011463 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.012450 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.012999 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.013066 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.013076 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.013502 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.015827 (+ 2325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260928 06:31:27.015914 32073 tablet_bootstrap.cc:654] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:27.016224 31780 tablet_service.cc:1467] Processing CreateTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 (DEFAULT_TABLE table=test-table [id=81f0b0e12f054a59a1b976d5c5c06359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:27.016352 31780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fce7addb0e943aaaa837b8e6eab7f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:27.016506 32073 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: No bootstrap required, opened a new log
I20260928 06:31:27.016573 32073 ts_tablet_manager.cc:1397] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:27.016692 32073 raft_consensus.cc:348] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.016779 32073 raft_consensus.cc:374] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:27.016811 32073 raft_consensus.cc:729] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fb529453174b2fb2ad5e088ef7e246, State: Initialized, Role: FOLLOWER
I20260928 06:31:27.016875 32073 consensus_queue.cc:260] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.017035 32073 ts_tablet_manager.cc:1428] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:27.017514 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.017281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:47767)
0928 06:31:27.017437 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.017478 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":90,"threads_started":1}
I20260928 06:31:27.017900 31711 heartbeater.cc:503] Master 127.24.11.189:34195 was elected leader, sending a full tablet report...
I20260928 06:31:27.017894 32078 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Bootstrap starting.
I20260928 06:31:27.018039 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.017738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:50855)
0928 06:31:27.017986 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.018014 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":75,"threads_started":1}
I20260928 06:31:27.018332 32078 tablet_bootstrap.cc:654] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:27.018967 32078 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: No bootstrap required, opened a new log
I20260928 06:31:27.018968 32072 tablet_bootstrap.cc:492] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: No bootstrap required, opened a new log
I20260928 06:31:27.019037 32078 ts_tablet_manager.cc:1397] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:27.019034 32072 ts_tablet_manager.cc:1397] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:31:27.019183 32078 raft_consensus.cc:348] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.019198 32072 raft_consensus.cc:348] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.019248 32072 raft_consensus.cc:374] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:27.019259 32078 raft_consensus.cc:374] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:27.019261 32072 raft_consensus.cc:729] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbb1b5f8d2a43b0b8fb98a277f8edce, State: Initialized, Role: FOLLOWER
I20260928 06:31:27.019284 32078 raft_consensus.cc:729] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1f7387082dc418c8f1d39e8f51d6086, State: Initialized, Role: FOLLOWER
I20260928 06:31:27.019333 32078 consensus_queue.cc:260] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.019330 32072 consensus_queue.cc:260] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.019479 32072 ts_tablet_manager.cc:1428] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:27.019572 32078 ts_tablet_manager.cc:1428] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:27.019858 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.019643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:51405)
0928 06:31:27.019780 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.019822 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260928 06:31:27.020169 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.019982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:35045)
0928 06:31:27.020104 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.020137 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":74,"threads_started":1}
I20260928 06:31:27.020537 31848 heartbeater.cc:503] Master 127.24.11.189:34195 was elected leader, sending a full tablet report...
I20260928 06:31:27.020543 31984 heartbeater.cc:503] Master 127.24.11.189:34195 was elected leader, sending a full tablet report...
I20260928 06:31:27.020525 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.020439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:51025)
0928 06:31:27.020473 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.020495 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:31:27.130923 32082 raft_consensus.cc:482] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:27.131016 32082 raft_consensus.cc:504] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.131428 32082 leader_election.cc:288] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971), e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
I20260928 06:31:27.135535 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.131764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:43033 (local address 127.24.11.131:39971)
0928 06:31:27.131806 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.131808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.131817 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:27.131848 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.131849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.131857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.131905 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:27.132111 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.132877 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.134493 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.135097 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.135182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.135193 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.135415 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.135465 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:27.135535 32071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.131757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:53571 (local address 127.24.11.129:36191)
0928 06:31:27.131822 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.131824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.131967 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:31:27.132034 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.132035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.132042 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.132111 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:27.132225 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.132919 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.134492 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.135097 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.135182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.135193 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.135415 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.135465 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:31:27.135535 32084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.131604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:39971 (local address 127.24.11.130:43033)
0928 06:31:27.131649 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.131663 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.131721 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.131937 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:27.131938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:27.131941 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:27.132029 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.132031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.132999 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.132999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.134369 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.134371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.135217 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.135219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.135312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.135316 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:27.135373 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:31:27.135427 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":77}
I20260928 06:31:27.135535 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.131664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36191 (local address 127.24.11.130:53571)
0928 06:31:27.131911 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.131922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.131975 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.132118 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:27.132119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:27.132120 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:27.132160 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.132161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.132995 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.132996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.134367 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:27.134371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.135221 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.135222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:27.135294 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.135300 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:31:27.135373 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:31:27.135427 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":75,"thread_start_us":119,"threads_started":1}
I20260928 06:31:27.135878 31663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fce7addb0e943aaaa837b8e6eab7f62" candidate_uuid: "b1f7387082dc418c8f1d39e8f51d6086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fb529453174b2fb2ad5e088ef7e246" is_pre_election: true
I20260928 06:31:27.136042 31663 raft_consensus.cc:2454] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f7387082dc418c8f1d39e8f51d6086 in term 0.
I20260928 06:31:27.136245 31936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fce7addb0e943aaaa837b8e6eab7f62" candidate_uuid: "b1f7387082dc418c8f1d39e8f51d6086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" is_pre_election: true
I20260928 06:31:27.136281 31733 leader_election.cc:302] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1f7387082dc418c8f1d39e8f51d6086, e1fb529453174b2fb2ad5e088ef7e246; no voters:
I20260928 06:31:27.136346 31936 raft_consensus.cc:2454] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1f7387082dc418c8f1d39e8f51d6086 in term 0.
I20260928 06:31:27.136425 32082 raft_consensus.cc:2781] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:27.136477 32082 raft_consensus.cc:482] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:27.136502 32082 raft_consensus.cc:3037] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:27.137348 32082 raft_consensus.cc:504] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.137560 32082 leader_election.cc:288] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [CANDIDATE]: Term 1 election: Requested vote from peers 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971), e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191)
I20260928 06:31:27.137710 31936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fce7addb0e943aaaa837b8e6eab7f62" candidate_uuid: "b1f7387082dc418c8f1d39e8f51d6086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce"
I20260928 06:31:27.137714 31663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fce7addb0e943aaaa837b8e6eab7f62" candidate_uuid: "b1f7387082dc418c8f1d39e8f51d6086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1fb529453174b2fb2ad5e088ef7e246"
I20260928 06:31:27.137789 31663 raft_consensus.cc:3037] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:27.137789 31936 raft_consensus.cc:3037] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:27.138543 31936 raft_consensus.cc:2454] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f7387082dc418c8f1d39e8f51d6086 in term 1.
I20260928 06:31:27.138631 31663 raft_consensus.cc:2454] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1f7387082dc418c8f1d39e8f51d6086 in term 1.
I20260928 06:31:27.138748 31734 leader_election.cc:302] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7dbb1b5f8d2a43b0b8fb98a277f8edce, b1f7387082dc418c8f1d39e8f51d6086; no voters:
I20260928 06:31:27.138851 32082 raft_consensus.cc:2781] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:27.139057 32082 raft_consensus.cc:686] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 1 LEADER]: Becoming Leader. State: Replica: b1f7387082dc418c8f1d39e8f51d6086, State: Running, Role: LEADER
I20260928 06:31:27.139147 32082 consensus_queue.cc:237] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } }
I20260928 06:31:27.139487 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.139257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:47565)
0928 06:31:27.139306 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.139459 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:31:27.139886 31343 catalog_manager.cc:5495] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "b1f7387082dc418c8f1d39e8f51d6086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1f7387082dc418c8f1d39e8f51d6086" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 43413 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:27.160014 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.157496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.24.11.130:43413)
0928 06:31:27.157608 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:27.157610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:27.157621 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:27.157687 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:27.157688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:27.157706 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:27.157771 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:27.157891 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.158400 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.159169 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:27.159360 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:27.159434 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:27.159594 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:27.159694 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:27.159752 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:27.159862 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:31:27.159883 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 06:31:27.161194 31936 raft_consensus.cc:1261] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Refusing update from remote peer b1f7387082dc418c8f1d39e8f51d6086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:27.161350 31663 raft_consensus.cc:1261] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Refusing update from remote peer b1f7387082dc418c8f1d39e8f51d6086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:27.161512 32082 consensus_queue.cc:1035] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dbb1b5f8d2a43b0b8fb98a277f8edce" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 39971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:27.161706 32086 consensus_queue.cc:1035] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1fb529453174b2fb2ad5e088ef7e246" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:27.162365 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.162226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:50881)
0928 06:31:27.162288 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.162323 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:31:27.162523 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.162385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:59883)
0928 06:31:27.162440 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.162475 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:31:27.164119 32085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.163800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.130:43485)
0928 06:31:27.164055 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.164087 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236}
I20260928 06:31:27.164165 32083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.164036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.131:54547)
0928 06:31:27.164106 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.164133 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 06:31:27.164202 32091 mvcc.cc:205] Tried to move back new op lower bound from 7334203749009936384 to 7334203748922314752. Current Snapshot: MvccSnapshot[applied={T|T < 7334203749009936384}]
I20260928 06:31:27.164407 32079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:27.164210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39467 (local address 127.24.11.129:49859)
0928 06:31:27.164350 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.164377 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:39467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 06:31:27.166450 31343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:27.166556 31343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:27.169723 31343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b545cd2ded54018a84389c63c1e4103: Sending DeleteTablet for 3 replicas of tablet 6fce7addb0e943aaaa837b8e6eab7f62
I20260928 06:31:27.170076 31643 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:27 UTC) from {username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:44078
I20260928 06:31:27.170181 31780 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:27 UTC) from {username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:34174
I20260928 06:31:27.170398 32100 tablet_replica.cc:330] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246: stopping tablet replica
I20260928 06:31:27.170415 32101 tablet_replica.cc:330] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: stopping tablet replica
I20260928 06:31:27.170486 32100 raft_consensus.cc:2229] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:27.170528 32101 raft_consensus.cc:2229] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:27.170601 32100 raft_consensus.cc:2258] T 6fce7addb0e943aaaa837b8e6eab7f62 P e1fb529453174b2fb2ad5e088ef7e246 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:27.170698 31916 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fce7addb0e943aaaa837b8e6eab7f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:27 UTC) from {username='oryx', principal='oryx/127.24.11.189@KRBTEST.COM'} at 127.0.0.1:51446
I20260928 06:31:27.170717 32101 raft_consensus.cc:2258] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:27.170982 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31491
I20260928 06:31:27.171128 32101 ts_tablet_manager.cc:1905] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:27.173103 32101 ts_tablet_manager.cc:1918] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:27.173192 32101 log.cc:1196] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/6fce7addb0e943aaaa837b8e6eab7f62
I20260928 06:31:27.173451 32101 ts_tablet_manager.cc:1939] T 6fce7addb0e943aaaa837b8e6eab7f62 P b1f7387082dc418c8f1d39e8f51d6086: Deleting consensus metadata
I20260928 06:31:27.173825 31328 catalog_manager.cc:4841] TS b1f7387082dc418c8f1d39e8f51d6086 (127.24.11.130:43413): tablet 6fce7addb0e943aaaa837b8e6eab7f62 (table test-table [id=81f0b0e12f054a59a1b976d5c5c06359]) successfully deleted
W20260928 06:31:27.177433 32068 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:27.177184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36191 (local address 127.0.0.1:44082)
0928 06:31:27.177246 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:27.177259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:27.177325 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:27.177381 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36191: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:31:27.177559 32102 tablet_replica.cc:330] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: stopping tablet replica
I20260928 06:31:27.177631 32102 raft_consensus.cc:2229] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:27.177685 32102 raft_consensus.cc:2258] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:31:27.177670 31328 catalog_manager.cc:4584] TS e1fb529453174b2fb2ad5e088ef7e246 (127.24.11.129:36191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fce7addb0e943aaaa837b8e6eab7f62: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36191: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 06:31:27.177862 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31717
I20260928 06:31:27.178890 32102 ts_tablet_manager.cc:1905] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:27.180351 32102 ts_tablet_manager.cc:1918] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:27.180414 32102 log.cc:1196] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/6fce7addb0e943aaaa837b8e6eab7f62
I20260928 06:31:27.180670 32102 ts_tablet_manager.cc:1939] T 6fce7addb0e943aaaa837b8e6eab7f62 P 7dbb1b5f8d2a43b0b8fb98a277f8edce: Deleting consensus metadata
I20260928 06:31:27.181020 31329 catalog_manager.cc:4841] TS 7dbb1b5f8d2a43b0b8fb98a277f8edce (127.24.11.131:39971): tablet 6fce7addb0e943aaaa837b8e6eab7f62 (table test-table [id=81f0b0e12f054a59a1b976d5c5c06359]) successfully deleted
I20260928 06:31:27.183897 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31853
I20260928 06:31:27.189199 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31211
I20260928 06:31:27.194280 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 31398
2026-09-28T06:31:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4782 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:27 dist-test-slave-38pg krb5kdc[32105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:27 dist-test-slave-38pg krb5kdc[32105](info): set up 2 sockets
Sep 28 06:31:27 dist-test-slave-38pg krb5kdc[32105](info): commencing operation
krb5kdc: starting...
W20260928 06:31:28.307368 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab.
Entry for principal oryx/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab.
I20260928 06:31:28.436547 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41639
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab
--principal=oryx/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:41639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:28.519152 32118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:28.519435 32118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:28.519474 32118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:28.520731 32118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:28.520810 32118 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:31:28.520835 32118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:28.520856 32118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:28.520877 32118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:28.520907 32118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:28.522634 32118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:41639
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.190.keytab
--principal=oryx/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:28.522941 32118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:28.523200 32118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:28.528486 32123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:28.528486 32126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:28.528504 32118 server_base.cc:1034] running on GCE node
W20260928 06:31:28.528498 32124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:28.529007 32118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:28.529295 32118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:28.530457 32118 hybrid_clock.cc:648] HybridClock initialized: now 1790577088530442 us; error 32 us; skew 500 ppm
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:28.540573 32118 init.cc:377] Logged in from keytab as oryx/127.24.11.190@KRBTEST.COM (short username oryx)
I20260928 06:31:28.541184 32118 webserver.cc:460] Webserver started at http://127.24.11.190:43333/ using document root <none> and password file <none>
I20260928 06:31:28.541460 32118 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:28.541535 32118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:28.541733 32118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:28.542840 32118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "451d0373a1164de8a9561e0c8b01f718"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "6233bf433cdb863e888ed6f569cc3699"
server_key_iv: "57ba43801205e2684111038fecff3083"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.543241 32118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "451d0373a1164de8a9561e0c8b01f718"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "6233bf433cdb863e888ed6f569cc3699"
server_key_iv: "57ba43801205e2684111038fecff3083"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.544652 32118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:28.545483 32133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.545734 32118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:28.545835 32118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "451d0373a1164de8a9561e0c8b01f718"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "6233bf433cdb863e888ed6f569cc3699"
server_key_iv: "57ba43801205e2684111038fecff3083"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.545893 32118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:28.572829 32118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:28.575500 32118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:28.575631 32118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:28.581655 32118 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:41639
I20260928 06:31:28.581758 32195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:41639 every 8 connection(s)
I20260928 06:31:28.582146 32118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:28.582964 32196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:28.585610 32196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Bootstrap starting.
I20260928 06:31:28.586297 32196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:28.586649 32196 log.cc:824] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:28.587576 32196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: No bootstrap required, opened a new log
I20260928 06:31:28.589869 32196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } }
I20260928 06:31:28.590031 32196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:28.590055 32196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451d0373a1164de8a9561e0c8b01f718, State: Initialized, Role: FOLLOWER
I20260928 06:31:28.590221 32196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } }
I20260928 06:31:28.590307 32196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:28.590332 32196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:28.590361 32196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:28.590986 32196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } }
I20260928 06:31:28.591077 32196 leader_election.cc:302] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 451d0373a1164de8a9561e0c8b01f718; no voters:
I20260928 06:31:28.591279 32196 leader_election.cc:288] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:28.591344 32199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:28.591578 32199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [term 1 LEADER]: Becoming Leader. State: Replica: 451d0373a1164de8a9561e0c8b01f718, State: Running, Role: LEADER
I20260928 06:31:28.591602 32196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:28.591676 32199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } }
I20260928 06:31:28.592115 32200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "451d0373a1164de8a9561e0c8b01f718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } } }
I20260928 06:31:28.592118 32201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 451d0373a1164de8a9561e0c8b01f718. Latest consensus state: current_term: 1 leader_uuid: "451d0373a1164de8a9561e0c8b01f718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "451d0373a1164de8a9561e0c8b01f718" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41639 } } }
I20260928 06:31:28.592253 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 32118
I20260928 06:31:28.592231 32200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:28.592238 32201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:28.592360 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:28.592516 32203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:28.592587 24622 external_mini_cluster.cc:1428] Setting key 4819956916f1ac14a2a4fcdf43e61cb3
I20260928 06:31:28.593096 32203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:28.594991 32203 catalog_manager.cc:1347] Generated new cluster ID: ca920db7bbff4af4bdbec055f0873420
I20260928 06:31:28.595064 32203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:28.599308 32203 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:28.599866 32203 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:28.605806 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.593261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.24.11.190:41639)
0928 06:31:28.593786 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:28.593797 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:28.593834 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:31:28.593986 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:28.593995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:28.594060 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:28.594211 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:28.595050 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.596522 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.597718 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.598409 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.598523 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.601688 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:28.601694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:28.601694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:28.601709 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:28.603911 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.604357 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.604361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.604362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.604516 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.604736 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.604736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.604737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.605078 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:28.605264 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:28.605348 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:31:28.605514 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":77,"threads_started":1}
I20260928 06:31:28.608811 32203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Generated new TSK 0
I20260928 06:31:28.609045 32203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
I20260928 06:31:28.639997 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41639
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:28.727977 32222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:28.728341 32222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:28.728394 32222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:28.730264 32222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:28.730322 32222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:28.730347 32222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:28.730367 32222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:28.732055 32222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:28.732287 32222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:28.732491 32222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:28.737907 32227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:28.738018 32228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:28.737933 32230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:28.738435 32222 server_base.cc:1034] running on GCE node
I20260928 06:31:28.738726 32222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:28.739033 32222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:28.740209 32222 hybrid_clock.cc:648] HybridClock initialized: now 1790577088740182 us; error 39 us; skew 500 ppm
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:28.743531 32222 init.cc:377] Logged in from keytab as oryx/127.24.11.129@KRBTEST.COM (short username oryx)
I20260928 06:31:28.744055 32222 webserver.cc:460] Webserver started at http://127.24.11.129:43169/ using document root <none> and password file <none>
I20260928 06:31:28.744302 32222 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:28.744351 32222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:28.744501 32222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:28.745767 32222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "0eebea79e0934c63985762b0a8926085"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "9187de357e3ca6812880f34f05a4ed07"
server_key_iv: "4eaad7043fdef007b4ef071964ff2419"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.746152 32222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "0eebea79e0934c63985762b0a8926085"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "9187de357e3ca6812880f34f05a4ed07"
server_key_iv: "4eaad7043fdef007b4ef071964ff2419"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.747651 32222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:28.748565 32237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.748878 32222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:28.748966 32222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "0eebea79e0934c63985762b0a8926085"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "9187de357e3ca6812880f34f05a4ed07"
server_key_iv: "4eaad7043fdef007b4ef071964ff2419"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.749073 32222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:28.762420 32222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:28.765059 32222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:28.765192 32222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:28.765946 32222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:28.765993 32222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.766043 32222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:28.766070 32222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:28.773571 32222 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:38691
I20260928 06:31:28.773723 32352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:38691 every 8 connection(s)
I20260928 06:31:28.774024 32222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:28.775509 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 32222
I20260928 06:31:28.775626 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:28.775861 24622 external_mini_cluster.cc:1428] Setting key bbadf41f54168cab02aad9652f8ec72d
I20260928 06:31:28.778833 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.766527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:43561 (local address 127.24.11.190:41639)
0928 06:31:28.766611 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:28.766614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:28.766826 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:31:28.767328 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:28.767329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:28.767344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:28.767438 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:28.768373 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.769279 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.770421 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.770811 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.770894 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.774108 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:28.774110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:28.774111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:28.774121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:28.776291 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.777625 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.777626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.777627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.777710 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.778052 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.778053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.778054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.778205 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:28.778296 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:28.778610 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 06:31:28.778726 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:31:28.779034 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.766504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.129:43561)
0928 06:31:28.766679 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.766721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:28.767224 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:28.767513 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:28.767516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:28.767916 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:28.768237 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.768244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.769388 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.769390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.770340 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.770343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.770940 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.770941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.771107 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.771815 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:28.771831 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:28.773981 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:28.776370 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.776378 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.776381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.776628 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.777801 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.777803 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.777803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.777953 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.778287 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:28.778298 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:28.778461 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:31:28.778588 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:31:28.778780 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260928 06:31:28.779716 32353 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41639
I20260928 06:31:28.779819 32353 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:28.780032 32353 heartbeater.cc:511] Master 127.24.11.190:41639 requested a full tablet report, sending...
I20260928 06:31:28.780682 32149 ts_manager.cc:194] Registered new tserver with Master: 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691)
I20260928 06:31:28.781759 32149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:43561
WARNING: no policy specified for oryx/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.130@KRBTEST.COM" created.
I20260928 06:31:28.787915 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.781655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:55387 (local address 127.24.11.190:41639)
0928 06:31:28.781700 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:28.781702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:28.781712 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:28.781746 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:28.781747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:28.781757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:28.781841 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:28.782292 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.782799 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.783541 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.783758 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.783904 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.784958 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:28.784959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:28.784959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:28.784967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:28.786570 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.787039 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.787040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.787041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.787102 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.787333 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.787334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.787334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.787478 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:28.787555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:28.787717 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:31:28.787813 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:31:28.787928 32360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.781009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.129:55387)
0928 06:31:28.781181 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.781196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:28.781249 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:28.781856 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:28.781874 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:28.782108 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:28.782201 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.782203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.782906 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.782907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.783447 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.783450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.783907 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.783908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.783983 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.784521 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:28.784534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:28.784836 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:28.786684 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.786685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.786686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.786933 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.787174 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.787175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.787176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.787249 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.787547 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:28.787548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:28.787601 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:31:28.787692 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:31:28.787812 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
I20260928 06:31:28.805730 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41639
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:28.890595 32362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:28.890826 32362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:28.890852 32362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:28.892203 32362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:28.892251 32362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:28.892273 32362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:28.892292 32362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:28.894066 32362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:28.894323 32362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:28.894584 32362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:28.899724 32368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:28.899729 32367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:28.899744 32370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:28.900110 32362 server_base.cc:1034] running on GCE node
I20260928 06:31:28.900380 32362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:28.900686 32362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:28.901885 32362 hybrid_clock.cc:648] HybridClock initialized: now 1790577088901864 us; error 34 us; skew 500 ppm
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:28.904552 32362 init.cc:377] Logged in from keytab as oryx/127.24.11.130@KRBTEST.COM (short username oryx)
I20260928 06:31:28.905023 32362 webserver.cc:460] Webserver started at http://127.24.11.130:34697/ using document root <none> and password file <none>
I20260928 06:31:28.905277 32362 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:28.905345 32362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:28.905521 32362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:28.906741 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "7b354b1abf3448eb81df079bd3c24d55"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "0d6f1fb790408d0f0d2dd4de4e3c1880"
server_key_iv: "e5cd3428527fef4317a994a2cc084dc6"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.907136 32362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "7b354b1abf3448eb81df079bd3c24d55"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "0d6f1fb790408d0f0d2dd4de4e3c1880"
server_key_iv: "e5cd3428527fef4317a994a2cc084dc6"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.908500 32362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:28.909230 32377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.909423 32362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:28.909518 32362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "7b354b1abf3448eb81df079bd3c24d55"
format_stamp: "Formatted at 2026-09-28 06:31:28 on dist-test-slave-38pg"
server_key: "0d6f1fb790408d0f0d2dd4de4e3c1880"
server_key_iv: "e5cd3428527fef4317a994a2cc084dc6"
server_key_version: "encryptionkey@0"
I20260928 06:31:28.909631 32362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:28.918598 32362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:28.921037 32362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:28.921157 32362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:28.921728 32362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:28.921772 32362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.921794 32362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:28.921808 32362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:28.929157 32362 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:39181
I20260928 06:31:28.929257 32492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:39181 every 8 connection(s)
I20260928 06:31:28.929682 32362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:31:28 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:28.931331 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 32362
I20260928 06:31:28.931489 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:28.931734 24622 external_mini_cluster.cc:1428] Setting key 2745359dba6aa7252707fef4641632aa
I20260928 06:31:28.934054 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.922221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:40129 (local address 127.24.11.190:41639)
0928 06:31:28.922298 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:28.922300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:28.922575 (+ 275us) server_negotiation.cc:407] Connection header received
0928 06:31:28.923426 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:28.923428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:28.923443 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:28.923527 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:28.924572 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.925489 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.927268 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.927516 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.927603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.930694 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:28.930696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:28.930696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:28.930704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:28.932406 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.932841 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.932842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.932842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.932913 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.933208 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.933209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.933210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.933371 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:28.933444 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:28.933806 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:31:28.933904 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:28.934145 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.922196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.130:40129)
0928 06:31:28.922409 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.922470 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:28.923243 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:28.923639 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:28.923644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:28.924096 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:28.924447 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.924455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.925642 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.925643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.927154 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.927157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.927610 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.927612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.927796 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.928565 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:28.928577 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:28.930567 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:28.932492 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.932500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.932503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.932760 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.933013 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.933014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.933015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.933132 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.933460 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:28.933466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:28.933668 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 06:31:28.933765 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:31:28.933950 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":102,"threads_started":1}
I20260928 06:31:28.934918 32493 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41639
I20260928 06:31:28.935019 32493 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:28.935233 32493 heartbeater.cc:511] Master 127.24.11.190:41639 requested a full tablet report, sending...
I20260928 06:31:28.935653 32149 ts_manager.cc:194] Registered new tserver with Master: 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:28.936038 32149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:40129
WARNING: no policy specified for oryx/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.131@KRBTEST.COM" created.
I20260928 06:31:28.947121 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.937984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35003 (local address 127.24.11.190:41639)
0928 06:31:28.938055 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:28.938056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:28.940805 (+ 2749us) server_negotiation.cc:407] Connection header received
0928 06:31:28.940842 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:28.940843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:28.940854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:28.940907 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:28.941241 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.941886 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.942605 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.942843 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.942898 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.944077 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:28.944078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:28.944079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:28.944089 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:28.945752 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.946204 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.946204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.946205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.946258 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:28.946494 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:28.946495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:28.946495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:28.946654 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:28.946715 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:28.946982 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:31:28.947056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:28.947209 32500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:28.937833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.130:35003)
0928 06:31:28.940693 (+ 2860us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:28.940708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:28.940791 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:28.940924 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:28.940924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:28.941106 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:28.941170 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.941172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.942008 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.942009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.942533 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:28.942534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:28.942917 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:28.942917 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:28.942971 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:28.943517 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:28.943523 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:28.943976 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:28.945846 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.945847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.945848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.946115 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.946323 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:28.946324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:28.946325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:28.946405 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:28.946730 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:28.946731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:28.946786 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:31:28.946915 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:31:28.947045 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2817,"thread_start_us":95,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
I20260928 06:31:28.960457 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41639
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:29.045744 32502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:29.045979 32502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:29.046008 32502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:29.047399 32502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:29.047453 32502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:29.047477 32502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:31:29.047529 32502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:29.049284 32502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.11.190:41639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:29.049613 32502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:29.049894 32502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:29.055362 32510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:29.055370 32508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:29.055352 32502 server_base.cc:1034] running on GCE node
W20260928 06:31:29.055349 32507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:29.055737 32502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:29.056010 32502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:29.057186 32502 hybrid_clock.cc:648] HybridClock initialized: now 1790577089057161 us; error 37 us; skew 500 ppm
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:29.059819 32502 init.cc:377] Logged in from keytab as oryx/127.24.11.131@KRBTEST.COM (short username oryx)
I20260928 06:31:29.060322 32502 webserver.cc:460] Webserver started at http://127.24.11.131:45147/ using document root <none> and password file <none>
I20260928 06:31:29.060588 32502 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:29.060668 32502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:29.060825 32502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:29.062119 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "bf1d60d2d5eb4d48b1f677393267cf49"
format_stamp: "Formatted at 2026-09-28 06:31:29 on dist-test-slave-38pg"
server_key: "518e6c3c70d051d78900e6b1ca1ed096"
server_key_iv: "ad600ff45037e3c10c5a8dd319858486"
server_key_version: "encryptionkey@0"
I20260928 06:31:29.062503 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "bf1d60d2d5eb4d48b1f677393267cf49"
format_stamp: "Formatted at 2026-09-28 06:31:29 on dist-test-slave-38pg"
server_key: "518e6c3c70d051d78900e6b1ca1ed096"
server_key_iv: "ad600ff45037e3c10c5a8dd319858486"
server_key_version: "encryptionkey@0"
I20260928 06:31:29.064011 32502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:29.064909 32517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.065121 32502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.065223 32502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "bf1d60d2d5eb4d48b1f677393267cf49"
format_stamp: "Formatted at 2026-09-28 06:31:29 on dist-test-slave-38pg"
server_key: "518e6c3c70d051d78900e6b1ca1ed096"
server_key_iv: "ad600ff45037e3c10c5a8dd319858486"
server_key_version: "encryptionkey@0"
I20260928 06:31:29.065347 32502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:29.082866 32502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:29.085613 32502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:29.085832 32502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:29.086505 32502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:29.086575 32502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.086621 32502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:29.086673 32502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:29.094594 32502 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:44659
I20260928 06:31:29.094692 32632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:44659 every 8 connection(s)
I20260928 06:31:29.095064 32502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:29.095927 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 32502
I20260928 06:31:29.096077 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:29.096303 24622 external_mini_cluster.cc:1428] Setting key 7ba446165afa7bfda32acc9be034fabc
I20260928 06:31:29.098228 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.087349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55019 (local address 127.24.11.190:41639)
0928 06:31:29.087403 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.087406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.087696 (+ 290us) server_negotiation.cc:407] Connection header received
0928 06:31:29.088262 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.088263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.088278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.088359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.089301 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.090219 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.091309 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.091523 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.091605 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.094845 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:29.094847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:29.094847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:29.094858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:29.096695 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.097254 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.097255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.097256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.097326 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.097666 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.097667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.097667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.097785 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:29.097844 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.098084 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.098163 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:29.098433 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.087350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.131:55019)
0928 06:31:29.087553 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.087591 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.088132 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.088469 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.088474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.088866 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:29.089186 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.089193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.090324 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.090325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.091189 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.091191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.091628 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.091629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.091784 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.092442 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:29.092464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:29.094685 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:29.096781 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.096787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.096789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.097170 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.097404 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.097405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.097405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.097601 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.097837 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:29.097840 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:29.097966 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.098050 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.098184 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":111,"threads_started":1}
I20260928 06:31:29.098912 32633 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41639
I20260928 06:31:29.099026 32633 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:29.099215 32633 heartbeater.cc:511] Master 127.24.11.190:41639 requested a full tablet report, sending...
I20260928 06:31:29.099591 32150 ts_manager.cc:194] Registered new tserver with Master: bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.099987 32150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:55019
I20260928 06:31:29.105337 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.100108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54459 (local address 127.24.11.190:41639)
0928 06:31:29.100217 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.100220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.100228 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:29.100285 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.100286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.100295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.100339 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.100660 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.101193 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.101936 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.102129 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.102186 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.103161 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:29.103162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:29.103163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:29.103169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:29.104391 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.104699 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.104700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.104700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.104751 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.104917 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.104918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.104918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.105027 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:29.105081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.105184 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.105271 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 06:31:29.105396 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.100047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.24.11.131:54459)
0928 06:31:29.100173 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.100185 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.100236 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.100357 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.100358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.100519 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:29.100576 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.100578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.101281 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.101281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.101862 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.101865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.102198 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.102199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.102249 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.102759 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:29.102771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:29.103081 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:29.104465 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.104466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.104466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.104634 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.104817 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.104817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.104818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.104866 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.105079 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:29.105079 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:29.105120 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.105169 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.105293 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 06:31:29.108359 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:31:29.131641 32212 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:29.125050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.24.11.190:41639)
0928 06:31:29.125153 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.125155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.125170 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:31:29.125232 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.125233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.125245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.125321 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.126091 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.126640 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.127639 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.127825 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.127890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.131511 (+ 3621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35944: BlockingRecv error: recv got EOF from 127.0.0.1:35944 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:29.140512 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.132603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.24.11.190:41639)
0928 06:31:29.132683 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.132685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.132750 (+ 65us) server_negotiation.cc:407] Connection header received
0928 06:31:29.132813 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.132813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.132824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.132883 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.133366 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.133936 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.134851 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.135072 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.135138 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.137658 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:29.137660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:29.137660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:29.137669 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:29.139155 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.139584 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.139586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.139586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.139660 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.139950 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.139951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.139951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.140102 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:29.140200 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.140353 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.140434 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:31:29.142771 32150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:29.144505 32150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:29.151741 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.24.11.129:38691)
0928 06:31:29.147692 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.147696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.147774 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:31:29.147889 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.147892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.147918 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.148008 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.148202 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149324 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.150737 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.151270 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.151381 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.151478 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.151583 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.151624 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":123,"threads_started":1}
I20260928 06:31:29.151770 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:38691 (local address 127.0.0.1:39874)
0928 06:31:29.147694 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.147703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.147784 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.148081 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.148082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.148084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.148128 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.148129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149420 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.149421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.150633 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.150636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.151386 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.151387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.151470 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.151488 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.151566 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.151622 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":49,"thread_start_us":82,"threads_started":1}
I20260928 06:31:29.151993 32655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:44659 (local address 127.0.0.1:56398)
0928 06:31:29.147178 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.147224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.147293 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.147882 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.147904 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.147907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.148006 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.148008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149401 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.149403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.150634 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.150637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.151722 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.151723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.151789 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.151794 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.151835 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.151882 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":90,"thread_start_us":93,"threads_started":1}
I20260928 06:31:29.152357 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.24.11.131:44659)
0928 06:31:29.147620 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.147627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.147642 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:31:29.147762 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.147765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.147783 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.147876 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.148096 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149281 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.150765 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.151528 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.151599 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.152090 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.152127 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.152259 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":133,"threads_started":1}
I20260928 06:31:29.152621 32287 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc8ccbdac24f60bf08944afe860595 (DEFAULT_TABLE table=test-table [id=19dadbae848e41c7b354657f1b7f6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.153007 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc8ccbdac24f60bf08944afe860595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.153126 32567 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc8ccbdac24f60bf08944afe860595 (DEFAULT_TABLE table=test-table [id=19dadbae848e41c7b354657f1b7f6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.153471 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc8ccbdac24f60bf08944afe860595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.154321 32658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39181 (local address 127.0.0.1:35968)
0928 06:31:29.147987 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.147997 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.148117 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.148366 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.148367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.148369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.148405 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.148406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149428 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.149429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.151785 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.151787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.154035 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.154036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.154099 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.154102 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.154156 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.154207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":92,"thread_start_us":88,"threads_started":1}
I20260928 06:31:29.154666 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.147418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.24.11.130:39181)
0928 06:31:29.147599 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.147604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.148093 (+ 489us) server_negotiation.cc:407] Connection header received
0928 06:31:29.148212 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.148215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.148229 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.148297 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.148487 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.149287 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.152929 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.153833 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.153900 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.153982 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.154525 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.154571 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260928 06:31:29.155375 32427 tablet_service.cc:1467] Processing CreateTablet for tablet c4cc8ccbdac24f60bf08944afe860595 (DEFAULT_TABLE table=test-table [id=19dadbae848e41c7b354657f1b7f6028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.155797 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4cc8ccbdac24f60bf08944afe860595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.157030 32661 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Bootstrap starting.
I20260928 06:31:29.157437 32662 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Bootstrap starting.
I20260928 06:31:29.157785 32661 tablet_bootstrap.cc:654] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.158106 32661 log.cc:824] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:29.158118 32662 tablet_bootstrap.cc:654] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.158483 32662 log.cc:824] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:29.159130 32661 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: No bootstrap required, opened a new log
I20260928 06:31:29.159235 32661 ts_tablet_manager.cc:1397] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.159330 32662 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: No bootstrap required, opened a new log
I20260928 06:31:29.159394 32662 ts_tablet_manager.cc:1397] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.159945 32665 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Bootstrap starting.
I20260928 06:31:29.160490 32665 tablet_bootstrap.cc:654] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.160799 32665 log.cc:824] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:29.161474 32665 tablet_bootstrap.cc:492] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: No bootstrap required, opened a new log
I20260928 06:31:29.161580 32665 ts_tablet_manager.cc:1397] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:29.161422 32661 raft_consensus.cc:348] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.161422 32662 raft_consensus.cc:348] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.161625 32661 raft_consensus.cc:374] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.161625 32662 raft_consensus.cc:374] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.161671 32661 raft_consensus.cc:729] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.161671 32662 raft_consensus.cc:729] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebea79e0934c63985762b0a8926085, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.161803 32661 consensus_queue.cc:260] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.161792 32662 consensus_queue.cc:260] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.162077 32662 ts_tablet_manager.cc:1428] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:29.162077 32661 ts_tablet_manager.cc:1428] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:29.162214 32353 heartbeater.cc:503] Master 127.24.11.190:41639 was elected leader, sending a full tablet report...
I20260928 06:31:29.162214 32633 heartbeater.cc:503] Master 127.24.11.190:41639 was elected leader, sending a full tablet report...
I20260928 06:31:29.163810 32665 raft_consensus.cc:348] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.163940 32665 raft_consensus.cc:374] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.163964 32665 raft_consensus.cc:729] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.164104 32665 consensus_queue.cc:260] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.164386 32665 ts_tablet_manager.cc:1428] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:31:29.164420 32493 heartbeater.cc:503] Master 127.24.11.190:41639 was elected leader, sending a full tablet report...
I20260928 06:31:29.169169 32667 raft_consensus.cc:482] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.169267 32667 raft_consensus.cc:504] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.169603 32667 leader_election.cc:288] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.172672 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.169817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:43655 (local address 127.24.11.129:38691)
0928 06:31:29.169893 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.169895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.169907 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:29.169936 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.169937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.169948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.170018 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.170174 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.170682 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.171782 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.172326 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.172383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.172390 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.172587 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.172617 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:31:29.172703 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.169739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:38691 (local address 127.24.11.131:43655)
0928 06:31:29.169797 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.169812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.169869 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.170035 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.170035 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.170037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.170113 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.170115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.170768 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.170769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.171690 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.171693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.172404 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.172405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.172512 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.172516 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.172577 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.172623 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":1}
I20260928 06:31:29.173092 32670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.169877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39181 (local address 127.24.11.131:49247)
0928 06:31:29.170081 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.170093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.170137 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.170337 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.170338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.170339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.170409 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.170411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.171207 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.171208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.172094 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.172096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.172836 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.172837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.172906 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.172909 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.172968 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.173008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":132,"threads_started":1}
I20260928 06:31:29.173116 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.170167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:49247 (local address 127.24.11.130:39181)
0928 06:31:29.170213 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.170214 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.170221 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:29.170260 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.170261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.170268 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.170333 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.170488 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.171117 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.172185 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.172702 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.172767 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.172776 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.173007 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.173051 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:29.173126 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc8ccbdac24f60bf08944afe860595" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085" is_pre_election: true
I20260928 06:31:29.173295 32307 raft_consensus.cc:2454] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.173381 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc8ccbdac24f60bf08944afe860595" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55" is_pre_election: true
I20260928 06:31:29.173558 32447 raft_consensus.cc:2454] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.173564 32520 leader_election.cc:302] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebea79e0934c63985762b0a8926085, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.173717 32667 raft_consensus.cc:2781] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:29.173764 32667 raft_consensus.cc:482] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:29.173799 32667 raft_consensus.cc:3037] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.174714 32667 raft_consensus.cc:504] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.174865 32667 leader_election.cc:288] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Requested vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.175051 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc8ccbdac24f60bf08944afe860595" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55"
I20260928 06:31:29.175163 32447 raft_consensus.cc:3037] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.175935 32447 raft_consensus.cc:2454] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.176074 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4cc8ccbdac24f60bf08944afe860595" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085"
I20260928 06:31:29.176153 32520 leader_election.cc:302] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.176170 32307 raft_consensus.cc:3037] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.176237 32667 raft_consensus.cc:2781] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:29.176404 32667 raft_consensus.cc:686] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 LEADER]: Becoming Leader. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Running, Role: LEADER
I20260928 06:31:29.176506 32667 consensus_queue.cc:237] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } }
I20260928 06:31:29.177160 32307 raft_consensus.cc:2454] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.177476 32150 catalog_manager.cc:5495] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } health_report { overall_health: UNKNOWN } } }
W20260928 06:31:29.180354 32494 tablet.cc:2369] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:29.187552 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.184882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.24.11.131:44659)
0928 06:31:29.184950 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.184952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.184982 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:31:29.185103 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.185104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.185123 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.185207 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.185366 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.185909 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.186716 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.186916 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.186994 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.187196 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.187301 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.187413 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.187433 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.187456 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:29.189678 32447 raft_consensus.cc:1261] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.189678 32307 raft_consensus.cc:1261] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.190117 32671 consensus_queue.cc:1035] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.190307 32667 consensus_queue.cc:1035] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.193391 32675 mvcc.cc:205] Tried to move back new op lower bound from 7334203757317246976 to 7334203757267755008. Current Snapshot: MvccSnapshot[applied={T|T < 7334203757317246976}]
I20260928 06:31:29.194126 32676 mvcc.cc:205] Tried to move back new op lower bound from 7334203757317246976 to 7334203757267755008. Current Snapshot: MvccSnapshot[applied={T|T < 7334203757317246976}]
I20260928 06:31:29.195501 32150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.195616 32150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.196455 32150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Sending DeleteTablet for 3 replicas of tablet c4cc8ccbdac24f60bf08944afe860595
I20260928 06:31:29.196828 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc8ccbdac24f60bf08944afe860595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35968
I20260928 06:31:29.196838 32287 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc8ccbdac24f60bf08944afe860595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:39874
I20260928 06:31:29.197090 32684 tablet_replica.cc:330] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: stopping tablet replica
I20260928 06:31:29.197147 32567 tablet_service.cc:1514] Processing DeleteTablet for tablet c4cc8ccbdac24f60bf08944afe860595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260928 06:31:29.197208 32684 raft_consensus.cc:2229] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.197378 32686 tablet_replica.cc:330] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: stopping tablet replica
I20260928 06:31:29.197372 32150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:29.197573 32150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:29.197661 32684 raft_consensus.cc:2258] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.197796 32686 raft_consensus.cc:2229] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:29.197995 32686 raft_consensus.cc:2258] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.198562 32686 ts_tablet_manager.cc:1905] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.198661 32684 ts_tablet_manager.cc:1905] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.199213 32427 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e25400ed4a0986c0adc82207a17b (DEFAULT_TABLE table=test-table [id=293f1871f0494dc5941ad7e1ba484123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.199360 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e25400ed4a0986c0adc82207a17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.199813 32685 tablet_replica.cc:330] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: stopping tablet replica
I20260928 06:31:29.200083 32685 raft_consensus.cc:2229] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.200214 32685 raft_consensus.cc:2258] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.200791 32685 ts_tablet_manager.cc:1905] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.200845 32665 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Bootstrap starting.
I20260928 06:31:29.201014 32686 ts_tablet_manager.cc:1918] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.201141 32686 log.cc:1196] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/c4cc8ccbdac24f60bf08944afe860595
I20260928 06:31:29.201278 32665 tablet_bootstrap.cc:654] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.201619 32686 ts_tablet_manager.cc:1939] T c4cc8ccbdac24f60bf08944afe860595 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting consensus metadata
I20260928 06:31:29.202096 32135 catalog_manager.cc:4841] TS bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659): tablet c4cc8ccbdac24f60bf08944afe860595 (table test-table [id=19dadbae848e41c7b354657f1b7f6028]) successfully deleted
I20260928 06:31:29.202116 32567 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e25400ed4a0986c0adc82207a17b (DEFAULT_TABLE table=test-table [id=293f1871f0494dc5941ad7e1ba484123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.202250 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e25400ed4a0986c0adc82207a17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.202903 32684 ts_tablet_manager.cc:1918] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.202992 32684 log.cc:1196] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/c4cc8ccbdac24f60bf08944afe860595
I20260928 06:31:29.203253 32661 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Bootstrap starting.
I20260928 06:31:29.203438 32287 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e25400ed4a0986c0adc82207a17b (DEFAULT_TABLE table=test-table [id=293f1871f0494dc5941ad7e1ba484123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.203557 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e25400ed4a0986c0adc82207a17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.203681 32661 tablet_bootstrap.cc:654] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.204377 32661 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: No bootstrap required, opened a new log
I20260928 06:31:29.204425 32661 ts_tablet_manager.cc:1397] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.204574 32661 raft_consensus.cc:348] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.204665 32661 raft_consensus.cc:374] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.204687 32661 raft_consensus.cc:729] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.204739 32661 consensus_queue.cc:260] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.204850 32661 ts_tablet_manager.cc:1428] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.204898 32662 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Bootstrap starting.
I20260928 06:31:29.205317 32685 ts_tablet_manager.cc:1918] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.205402 32685 log.cc:1196] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/c4cc8ccbdac24f60bf08944afe860595
I20260928 06:31:29.205679 32684 ts_tablet_manager.cc:1939] T c4cc8ccbdac24f60bf08944afe860595 P 0eebea79e0934c63985762b0a8926085: Deleting consensus metadata
I20260928 06:31:29.205734 32685 ts_tablet_manager.cc:1939] T c4cc8ccbdac24f60bf08944afe860595 P 7b354b1abf3448eb81df079bd3c24d55: Deleting consensus metadata
I20260928 06:31:29.205742 32662 tablet_bootstrap.cc:654] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.206274 32136 catalog_manager.cc:4841] TS 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691): tablet c4cc8ccbdac24f60bf08944afe860595 (table test-table [id=19dadbae848e41c7b354657f1b7f6028]) successfully deleted
I20260928 06:31:29.206393 32136 catalog_manager.cc:4841] TS 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181): tablet c4cc8ccbdac24f60bf08944afe860595 (table test-table [id=19dadbae848e41c7b354657f1b7f6028]) successfully deleted
I20260928 06:31:29.206439 32662 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: No bootstrap required, opened a new log
I20260928 06:31:29.206489 32662 ts_tablet_manager.cc:1397] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.206590 32665 tablet_bootstrap.cc:492] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: No bootstrap required, opened a new log
I20260928 06:31:29.206651 32665 ts_tablet_manager.cc:1397] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 06:31:29.206650 32662 raft_consensus.cc:348] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.206738 32662 raft_consensus.cc:374] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.206770 32662 raft_consensus.cc:729] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebea79e0934c63985762b0a8926085, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.206832 32662 consensus_queue.cc:260] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.206825 32665 raft_consensus.cc:348] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.206897 32665 raft_consensus.cc:374] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.206926 32665 raft_consensus.cc:729] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.206931 32662 ts_tablet_manager.cc:1428] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.206980 32665 consensus_queue.cc:260] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.207077 32665 ts_tablet_manager.cc:1428] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 06:31:29.274927 32354 tablet.cc:2369] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:29.325958 32669 raft_consensus.cc:482] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.326063 32669 raft_consensus.cc:504] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.326474 32669 leader_election.cc:288] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.329392 32668 raft_consensus.cc:482] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.329571 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.326676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:38691 (local address 127.24.11.130:36411)
0928 06:31:29.326744 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.326758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.326828 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.327036 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.327037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.327039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.327120 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.327122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.327804 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.327805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.328647 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.328650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.329241 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.329242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.329328 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.329332 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.329406 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.329455 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:31:29.329483 32668 raft_consensus.cc:504] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.329644 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.326760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:36411 (local address 127.24.11.129:38691)
0928 06:31:29.326807 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.326809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.326820 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:29.326953 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.326954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.326963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.327033 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.327198 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.327704 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.328759 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.329151 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.329241 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.329251 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.329416 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.329473 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:29.329674 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.326772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:51413 (local address 127.24.11.131:44659)
0928 06:31:29.326822 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.326824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.326943 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:31:29.327053 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.327055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.327066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.327096 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.327305 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.327765 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.328796 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.329190 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.329319 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.329329 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.329519 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.329609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:29.329895 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.326692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:44659 (local address 127.24.11.130:51413)
0928 06:31:29.326847 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.326857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.326974 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.327160 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.327160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.327162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.327232 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.327233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.327870 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.327871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.328690 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.328692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.329338 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.329339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.329425 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.329429 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.329744 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.329792 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260928 06:31:29.329929 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085" is_pre_election: true
I20260928 06:31:29.329972 32668 leader_election.cc:288] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181), bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.330008 32307 raft_consensus.cc:2454] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 0.
I20260928 06:31:29.330283 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" is_pre_election: true
I20260928 06:31:29.330453 32587 raft_consensus.cc:2454] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 0.
I20260928 06:31:29.330673 32379 leader_election.cc:302] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.330835 32669 raft_consensus.cc:2781] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:29.330874 32669 raft_consensus.cc:482] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:29.330902 32669 raft_consensus.cc:3037] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.331836 32669 raft_consensus.cc:504] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.331992 32669 leader_election.cc:288] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 election: Requested vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.332168 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1d60d2d5eb4d48b1f677393267cf49"
I20260928 06:31:29.332173 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085"
I20260928 06:31:29.332232 32587 raft_consensus.cc:3037] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.332237 32307 raft_consensus.cc:3037] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.332962 32307 raft_consensus.cc:2454] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 1.
I20260928 06:31:29.333086 32587 raft_consensus.cc:2454] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 1.
I20260928 06:31:29.333142 32380 leader_election.cc:302] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebea79e0934c63985762b0a8926085, 7b354b1abf3448eb81df079bd3c24d55; no voters:
I20260928 06:31:29.333221 32669 raft_consensus.cc:2781] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:29.333379 32669 raft_consensus.cc:686] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Running, Role: LEADER
I20260928 06:31:29.333457 32669 consensus_queue.cc:237] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.333645 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.330275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:42883 (local address 127.24.11.130:39181)
0928 06:31:29.330322 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.330325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.330334 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:29.330411 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.330412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.330422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.330450 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.330662 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.331306 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.332661 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.333225 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.333267 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.333275 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.333523 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.333581 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:31:29.333899 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.330097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39181 (local address 127.24.11.129:42883)
0928 06:31:29.330267 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.330279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.330330 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.330510 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.330511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.330519 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.330584 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.330585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.331387 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.331388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.332563 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.332565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.333346 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.333347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.333444 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.333447 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.333754 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.333806 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260928 06:31:29.334072 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.330365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:48443 (local address 127.24.11.131:44659)
0928 06:31:29.330397 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.330399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.330801 (+ 402us) server_negotiation.cc:407] Connection header received
0928 06:31:29.330891 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.330892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.330901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.330942 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:29.331164 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.331823 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.333139 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.333690 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.333738 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.333747 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.333970 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.334017 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:29.334111 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "0eebea79e0934c63985762b0a8926085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55" is_pre_election: true
I20260928 06:31:29.334262 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.330119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:44659 (local address 127.24.11.129:48443)
0928 06:31:29.330707 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.330735 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.330827 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.331034 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.331035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.331036 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:29.331091 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.331092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.331925 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.331927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.333047 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.333049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.333803 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.333804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.333881 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.333884 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.334143 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.334193 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":53,"threads_started":1}
I20260928 06:31:29.334360 32147 catalog_manager.cc:5495] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "7b354b1abf3448eb81df079bd3c24d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:29.334899 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e25400ed4a0986c0adc82207a17b" candidate_uuid: "0eebea79e0934c63985762b0a8926085" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" is_pre_election: true
I20260928 06:31:29.335021 32587 raft_consensus.cc:2379] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0eebea79e0934c63985762b0a8926085 in current term 1: Already voted for candidate 7b354b1abf3448eb81df079bd3c24d55 in this term.
I20260928 06:31:29.335222 32239 leader_election.cc:302] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0eebea79e0934c63985762b0a8926085; no voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49
I20260928 06:31:29.335419 32668 raft_consensus.cc:2726] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260928 06:31:29.345649 32634 tablet.cc:2369] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577088, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.190@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:29.374217 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.366417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41639 (local address 127.0.0.1:35954)
0928 06:31:29.366666 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.366678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.366785 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.366952 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.366953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.367358 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:29.367414 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.367417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.368131 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.368132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.368775 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.368777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.369141 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.369142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.369192 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.369872 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:29.369877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:29.371300 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:29.372945 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.372955 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.372958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.373157 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.373390 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:29.373391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:29.373391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:29.373461 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:29.373744 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:29.373753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:29.373852 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.373927 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.374054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":137,"threads_started":1}
I20260928 06:31:29.374217 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.366515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.24.11.190:41639)
0928 06:31:29.366639 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.366642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.366763 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:31:29.366849 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.366850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.366862 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.366944 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.367521 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.368033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.368865 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.369050 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.369140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.371394 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:29.371396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:29.371396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:29.371404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:29.372844 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.373247 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.373248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.373249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.373306 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.373556 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.373557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.373557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.373676 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:29.373753 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.373974 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.374103 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":90}
I20260928 06:31:29.376732 32147 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35954:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:31:29.376937 32147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:29.378629 32287 tablet_service.cc:1467] Processing CreateTablet for tablet f4d098497703400e80f32beb8b1d5ca7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a69c09a135144669d6b5bd7e68bf8ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:29.378612 32427 tablet_service.cc:1467] Processing CreateTablet for tablet f4d098497703400e80f32beb8b1d5ca7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a69c09a135144669d6b5bd7e68bf8ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:29.378641 32567 tablet_service.cc:1467] Processing CreateTablet for tablet f4d098497703400e80f32beb8b1d5ca7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2a69c09a135144669d6b5bd7e68bf8ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:31:29.378791 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d098497703400e80f32beb8b1d5ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.378790 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d098497703400e80f32beb8b1d5ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.378808 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d098497703400e80f32beb8b1d5ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.380229 32661 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: Bootstrap starting.
I20260928 06:31:29.380331 32662 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: Bootstrap starting.
I20260928 06:31:29.380324 32665 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: Bootstrap starting.
I20260928 06:31:29.380648 32661 tablet_bootstrap.cc:654] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.380810 32662 tablet_bootstrap.cc:654] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.380827 32665 tablet_bootstrap.cc:654] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.381554 32665 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: No bootstrap required, opened a new log
I20260928 06:31:29.381623 32662 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: No bootstrap required, opened a new log
I20260928 06:31:29.381631 32665 ts_tablet_manager.cc:1397] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.381677 32662 ts_tablet_manager.cc:1397] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.381839 32662 raft_consensus.cc:348] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.381935 32662 raft_consensus.cc:374] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.381956 32662 raft_consensus.cc:729] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebea79e0934c63985762b0a8926085, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.381992 32665 raft_consensus.cc:348] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.382017 32662 consensus_queue.cc:260] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.382066 32665 raft_consensus.cc:374] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.382093 32665 raft_consensus.cc:729] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.382114 32662 ts_tablet_manager.cc:1428] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.382141 32665 consensus_queue.cc:260] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.382174 32668 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.382243 32665 ts_tablet_manager.cc:1428] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.382272 32668 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.382313 32693 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.382395 32693 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.382647 32661 tablet_bootstrap.cc:492] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: No bootstrap required, opened a new log
I20260928 06:31:29.382701 32661 ts_tablet_manager.cc:1397] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.382840 32661 raft_consensus.cc:348] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.382934 32661 raft_consensus.cc:374] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.382967 32661 raft_consensus.cc:729] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.383020 32661 consensus_queue.cc:260] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.383112 32661 ts_tablet_manager.cc:1428] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.383159 32681 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.383256 32681 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.456184 32681 raft_consensus.cc:482] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.456283 32681 raft_consensus.cc:504] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.456450 32681 leader_election.cc:288] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.456708 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d098497703400e80f32beb8b1d5ca7" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085" is_pre_election: true
I20260928 06:31:29.456766 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d098497703400e80f32beb8b1d5ca7" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55" is_pre_election: true
I20260928 06:31:29.456838 32307 raft_consensus.cc:2454] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.456848 32447 raft_consensus.cc:2454] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.456997 32520 leader_election.cc:302] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.457103 32681 raft_consensus.cc:2781] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:29.457158 32681 raft_consensus.cc:482] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:29.457190 32681 raft_consensus.cc:3037] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.457973 32681 raft_consensus.cc:504] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.458099 32681 leader_election.cc:288] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Requested vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.458313 32307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d098497703400e80f32beb8b1d5ca7" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085"
I20260928 06:31:29.458388 32307 raft_consensus.cc:3037] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.458331 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d098497703400e80f32beb8b1d5ca7" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55"
I20260928 06:31:29.458422 32447 raft_consensus.cc:3037] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.458968 32307 raft_consensus.cc:2454] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.459137 32520 leader_election.cc:302] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebea79e0934c63985762b0a8926085, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.459157 32447 raft_consensus.cc:2454] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.459280 32681 raft_consensus.cc:2781] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:29.459345 32681 raft_consensus.cc:686] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 LEADER]: Becoming Leader. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Running, Role: LEADER
I20260928 06:31:29.459401 32681 consensus_queue.cc:237] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.459661 32667 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: TxnStatusTablet state changed. Reason: New leader bf1d60d2d5eb4d48b1f677393267cf49. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.459748 32667 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:29.459962 32705 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:31:29.460083 32705 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:31:29.460036 32150 catalog_manager.cc:5495] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:29.478628 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.476140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.24.11.131:44659)
0928 06:31:29.476211 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.476213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.476223 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:29.476331 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.476332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.476344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.476402 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.476561 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.477038 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.477878 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.478087 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.478162 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.478342 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.478384 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.478446 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.478521 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.478543 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:29.478629 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.476054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:44659 (local address 127.0.0.1:56406)
0928 06:31:29.476160 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.476171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.476223 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.476426 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.476427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.476429 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:29.476484 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.476486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.477130 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.477131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.477791 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.477794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.478173 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.478174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.478234 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.478249 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.478449 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.478459 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.478510 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.478558 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:31:29.479642 32307 raft_consensus.cc:1261] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.479755 32447 raft_consensus.cc:1261] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.479934 32681 consensus_queue.cc:1035] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.480032 32667 consensus_queue.cc:1035] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.480615 32693 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: TxnStatusTablet state changed. Reason: New leader bf1d60d2d5eb4d48b1f677393267cf49. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.480702 32693 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.481032 32668 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: TxnStatusTablet state changed. Reason: New leader bf1d60d2d5eb4d48b1f677393267cf49. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.481128 32668 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.481389 32674 mvcc.cc:205] Tried to move back new op lower bound from 7334203758506737664 to 7334203758425980928. Current Snapshot: MvccSnapshot[applied={T|T < 7334203758506737664}]
I20260928 06:31:29.481457 32667 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.481554 32667 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:29.482168 32693 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.482261 32693 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 7b354b1abf3448eb81df079bd3c24d55: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.483019 32681 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.483104 32681 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P bf1d60d2d5eb4d48b1f677393267cf49: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:31:29.483246 32668 tablet_replica.cc:439] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } } }
I20260928 06:31:29.483332 32668 tablet_replica.cc:442] T f4d098497703400e80f32beb8b1d5ca7 P 0eebea79e0934c63985762b0a8926085: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:31:29.485978 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.483366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.24.11.130:39181)
0928 06:31:29.483414 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.483416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.483478 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:31:29.483602 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.483603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.483626 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.483705 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.483868 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.484387 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.485125 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.485333 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.485414 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.485605 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.485721 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.485810 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.485849 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.485877 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:31:29.491092 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.488453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:55469 (local address 127.24.11.131:44659)
0928 06:31:29.488511 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.488513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.488606 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:31:29.488680 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.488680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.488689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.488733 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.488870 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.489298 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.490407 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.490610 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.490669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.490847 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.490904 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.490943 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.491020 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.491038 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:31:29.491144 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.488358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:44659 (local address 127.24.11.130:55469)
0928 06:31:29.488550 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.488562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.488624 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.488738 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.488739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.488753 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:29.488809 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.488810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.489370 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.489370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.490313 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.490315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.490687 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.490688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.490754 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.490768 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.490963 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.490966 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.491020 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.491063 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":100,"threads_started":1}
I20260928 06:31:29.493656 32307 raft_consensus.cc:1261] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Refusing update from remote peer 7b354b1abf3448eb81df079bd3c24d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.493705 32587 raft_consensus.cc:1261] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Refusing update from remote peer 7b354b1abf3448eb81df079bd3c24d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.493942 32669 consensus_queue.cc:1035] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.494133 32693 consensus_queue.cc:1035] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.502746 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.499896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39181 (local address 127.24.11.131:53001)
0928 06:31:29.499953 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.499967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:29.500008 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:29.500132 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:29.500132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:29.500133 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:31:29.500183 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.500184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.501046 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.501047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.501891 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:29.501894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.502373 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.502373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:29.502421 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.502435 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.502614 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.502616 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:29.502651 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 06:31:29.502691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:31:29.502746 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.499992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:53001 (local address 127.24.11.130:39181)
0928 06:31:29.500028 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.500030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.500037 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:31:29.500058 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.500059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.500068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.500128 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.500228 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.500962 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.501964 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.502276 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.502346 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.502520 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.502554 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.502606 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.502654 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.502673 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:29.512027 32147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.512152 32147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.512807 32147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Sending DeleteTablet for 3 replicas of tablet 9e82e25400ed4a0986c0adc82207a17b
I20260928 06:31:29.513047 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e25400ed4a0986c0adc82207a17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35968
I20260928 06:31:29.513067 32287 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e25400ed4a0986c0adc82207a17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:39874
I20260928 06:31:29.513144 32685 tablet_replica.cc:330] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: stopping tablet replica
I20260928 06:31:29.513154 32684 tablet_replica.cc:330] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: stopping tablet replica
I20260928 06:31:29.513190 32685 raft_consensus.cc:2229] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:29.513198 32684 raft_consensus.cc:2229] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.513234 32684 raft_consensus.cc:2258] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.513273 32685 raft_consensus.cc:2258] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.513653 32567 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e25400ed4a0986c0adc82207a17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260928 06:31:29.513752 32684 ts_tablet_manager.cc:1905] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.513760 32686 tablet_replica.cc:330] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: stopping tablet replica
I20260928 06:31:29.513834 32686 raft_consensus.cc:2229] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.513868 32686 raft_consensus.cc:2258] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.513993 32685 ts_tablet_manager.cc:1905] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.514153 32686 ts_tablet_manager.cc:1905] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.515725 32684 ts_tablet_manager.cc:1918] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:29.515793 32684 log.cc:1196] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/9e82e25400ed4a0986c0adc82207a17b
I20260928 06:31:29.515874 32686 ts_tablet_manager.cc:1918] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:29.515924 32686 log.cc:1196] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/9e82e25400ed4a0986c0adc82207a17b
I20260928 06:31:29.516019 32684 ts_tablet_manager.cc:1939] T 9e82e25400ed4a0986c0adc82207a17b P 0eebea79e0934c63985762b0a8926085: Deleting consensus metadata
I20260928 06:31:29.516124 32686 ts_tablet_manager.cc:1939] T 9e82e25400ed4a0986c0adc82207a17b P bf1d60d2d5eb4d48b1f677393267cf49: Deleting consensus metadata
I20260928 06:31:29.516428 32136 catalog_manager.cc:4841] TS 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691): tablet 9e82e25400ed4a0986c0adc82207a17b (table test-table [id=293f1871f0494dc5941ad7e1ba484123]) successfully deleted
I20260928 06:31:29.516522 32135 catalog_manager.cc:4841] TS bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659): tablet 9e82e25400ed4a0986c0adc82207a17b (table test-table [id=293f1871f0494dc5941ad7e1ba484123]) successfully deleted
I20260928 06:31:29.519238 32685 ts_tablet_manager.cc:1918] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:29.519307 32685 log.cc:1196] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/9e82e25400ed4a0986c0adc82207a17b
I20260928 06:31:29.519557 32685 ts_tablet_manager.cc:1939] T 9e82e25400ed4a0986c0adc82207a17b P 7b354b1abf3448eb81df079bd3c24d55: Deleting consensus metadata
I20260928 06:31:29.519918 32136 catalog_manager.cc:4841] TS 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181): tablet 9e82e25400ed4a0986c0adc82207a17b (table test-table [id=293f1871f0494dc5941ad7e1ba484123]) successfully deleted
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:31:29.552556 32212 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:29.545143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.24.11.190:41639)
0928 06:31:29.545196 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.545198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.545428 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:31:29.545472 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.545474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.545487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.545594 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.546335 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.547090 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.548195 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.548500 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.548586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.552487 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM
I20260928 06:31:29.567092 32212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.553767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.24.11.190:41639)
0928 06:31:29.553808 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.553810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.557798 (+ 3988us) server_negotiation.cc:407] Connection header received
0928 06:31:29.557847 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.557848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.557862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.557939 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:29.558611 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.559376 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.560486 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.560790 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.560873 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.563856 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:29.563858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:29.563858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:29.563867 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:29.565748 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.566175 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.566176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.566177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.566243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:29.566491 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:29.566492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:29.566492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:29.566640 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:29.566743 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.566901 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.566991 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:29.569270 32147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:29.569453 32147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:29.570833 32567 tablet_service.cc:1467] Processing CreateTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 (DEFAULT_TABLE table=test-table [id=42b4ed38ab0c4976bac4781a6efa234e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.570889 32287 tablet_service.cc:1467] Processing CreateTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 (DEFAULT_TABLE table=test-table [id=42b4ed38ab0c4976bac4781a6efa234e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.570974 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e48bdf75a49d0b597b9ec60cdacf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.571025 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e48bdf75a49d0b597b9ec60cdacf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.572299 32661 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Bootstrap starting.
I20260928 06:31:29.572415 32427 tablet_service.cc:1467] Processing CreateTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 (DEFAULT_TABLE table=test-table [id=42b4ed38ab0c4976bac4781a6efa234e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.572547 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 089e48bdf75a49d0b597b9ec60cdacf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.572657 32661 tablet_bootstrap.cc:654] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.573858 32662 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Bootstrap starting.
I20260928 06:31:29.574110 32661 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: No bootstrap required, opened a new log
I20260928 06:31:29.574156 32661 ts_tablet_manager.cc:1397] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.574266 32661 raft_consensus.cc:348] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.574324 32661 raft_consensus.cc:374] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.574342 32661 raft_consensus.cc:729] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.574393 32661 consensus_queue.cc:260] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.574476 32661 ts_tablet_manager.cc:1428] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.574508 32662 tablet_bootstrap.cc:654] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.574864 32665 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Bootstrap starting.
I20260928 06:31:29.575253 32665 tablet_bootstrap.cc:654] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.577229 32665 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: No bootstrap required, opened a new log
I20260928 06:31:29.577278 32665 ts_tablet_manager.cc:1397] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.577412 32665 raft_consensus.cc:348] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.577492 32665 raft_consensus.cc:374] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.577800 32662 tablet_bootstrap.cc:492] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: No bootstrap required, opened a new log
I20260928 06:31:29.577848 32662 ts_tablet_manager.cc:1397] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 06:31:29.577983 32662 raft_consensus.cc:348] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.578043 32662 raft_consensus.cc:374] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.578060 32662 raft_consensus.cc:729] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebea79e0934c63985762b0a8926085, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.578109 32662 consensus_queue.cc:260] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.578189 32662 ts_tablet_manager.cc:1428] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.577510 32665 raft_consensus.cc:729] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.578414 32665 consensus_queue.cc:260] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.578526 32665 ts_tablet_manager.cc:1428] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:29.659400 32671 raft_consensus.cc:482] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.659518 32671 raft_consensus.cc:504] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.659649 32671 leader_election.cc:288] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.660012 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e48bdf75a49d0b597b9ec60cdacf4" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085" is_pre_election: true
I20260928 06:31:29.659988 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e48bdf75a49d0b597b9ec60cdacf4" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55" is_pre_election: true
I20260928 06:31:29.660104 32447 raft_consensus.cc:2454] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.660102 32306 raft_consensus.cc:2454] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 0.
I20260928 06:31:29.660269 32520 leader_election.cc:302] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.660386 32671 raft_consensus.cc:2781] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:29.660440 32671 raft_consensus.cc:482] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:29.660471 32671 raft_consensus.cc:3037] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.661175 32671 raft_consensus.cc:504] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.661341 32671 leader_election.cc:288] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Requested vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181)
I20260928 06:31:29.661582 32447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e48bdf75a49d0b597b9ec60cdacf4" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b354b1abf3448eb81df079bd3c24d55"
I20260928 06:31:29.661579 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "089e48bdf75a49d0b597b9ec60cdacf4" candidate_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085"
I20260928 06:31:29.661679 32306 raft_consensus.cc:3037] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.661679 32447 raft_consensus.cc:3037] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.662492 32306 raft_consensus.cc:2454] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.662492 32447 raft_consensus.cc:2454] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf1d60d2d5eb4d48b1f677393267cf49 in term 1.
I20260928 06:31:29.662683 32520 leader_election.cc:302] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.662839 32671 raft_consensus.cc:2781] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:29.662899 32671 raft_consensus.cc:686] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 LEADER]: Becoming Leader. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Running, Role: LEADER
I20260928 06:31:29.662947 32671 consensus_queue.cc:237] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.663766 32147 catalog_manager.cc:5495] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } health_report { overall_health: HEALTHY } } }
I20260928 06:31:29.676318 32657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.674011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.24.11.131:44659)
0928 06:31:29.674055 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.674057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.674066 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:29.674094 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.674095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.674111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.674174 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.674326 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.674804 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.675578 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.675780 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.675867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.676053 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.676090 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.676152 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.676214 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.676233 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:31:29.677186 32447 raft_consensus.cc:1261] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.677198 32306 raft_consensus.cc:1261] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Refusing update from remote peer bf1d60d2d5eb4d48b1f677393267cf49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.677453 32681 consensus_queue.cc:1035] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.677630 32671 consensus_queue.cc:1035] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.680121 32147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.680233 32147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.680896 32147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Sending DeleteTablet for 3 replicas of tablet 089e48bdf75a49d0b597b9ec60cdacf4
I20260928 06:31:29.681124 32567 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260928 06:31:29.681138 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35968
I20260928 06:31:29.681156 32287 tablet_service.cc:1514] Processing DeleteTablet for tablet 089e48bdf75a49d0b597b9ec60cdacf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:39874
I20260928 06:31:29.681226 32685 tablet_replica.cc:330] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: stopping tablet replica
I20260928 06:31:29.681242 32684 tablet_replica.cc:330] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: stopping tablet replica
I20260928 06:31:29.681277 32685 raft_consensus.cc:2229] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.681283 32684 raft_consensus.cc:2229] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.681327 32684 raft_consensus.cc:2258] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.681349 32685 raft_consensus.cc:2258] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.681427 32686 tablet_replica.cc:330] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: stopping tablet replica
I20260928 06:31:29.681485 32686 raft_consensus.cc:2229] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:29.681445 32147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:31:29.681591 32686 raft_consensus.cc:2258] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:31:29.681622 32147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:29.682008 32685 ts_tablet_manager.cc:1905] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.682075 32686 ts_tablet_manager.cc:1905] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.683296 32684 ts_tablet_manager.cc:1905] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.683498 32685 ts_tablet_manager.cc:1918] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.683559 32685 log.cc:1196] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/089e48bdf75a49d0b597b9ec60cdacf4
I20260928 06:31:29.683786 32685 ts_tablet_manager.cc:1939] T 089e48bdf75a49d0b597b9ec60cdacf4 P 7b354b1abf3448eb81df079bd3c24d55: Deleting consensus metadata
I20260928 06:31:29.684088 32136 catalog_manager.cc:4841] TS 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181): tablet 089e48bdf75a49d0b597b9ec60cdacf4 (table test-table [id=42b4ed38ab0c4976bac4781a6efa234e]) successfully deleted
I20260928 06:31:29.684713 32684 ts_tablet_manager.cc:1918] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.684712 32686 ts_tablet_manager.cc:1918] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:31:29.684772 32684 log.cc:1196] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/089e48bdf75a49d0b597b9ec60cdacf4
I20260928 06:31:29.684772 32686 log.cc:1196] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/089e48bdf75a49d0b597b9ec60cdacf4
I20260928 06:31:29.684990 32684 ts_tablet_manager.cc:1939] T 089e48bdf75a49d0b597b9ec60cdacf4 P 0eebea79e0934c63985762b0a8926085: Deleting consensus metadata
I20260928 06:31:29.685003 32686 ts_tablet_manager.cc:1939] T 089e48bdf75a49d0b597b9ec60cdacf4 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting consensus metadata
I20260928 06:31:29.685024 32287 tablet_service.cc:1467] Processing CreateTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 (DEFAULT_TABLE table=test-table [id=df191cd9f3a341488849a95bed09ea3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.685134 32287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad07e9ccfae54f13a96a5acb76f395e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.685259 32567 tablet_service.cc:1467] Processing CreateTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 (DEFAULT_TABLE table=test-table [id=df191cd9f3a341488849a95bed09ea3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.685261 32427 tablet_service.cc:1467] Processing CreateTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 (DEFAULT_TABLE table=test-table [id=df191cd9f3a341488849a95bed09ea3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:29.685350 32427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad07e9ccfae54f13a96a5acb76f395e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.685353 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad07e9ccfae54f13a96a5acb76f395e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:29.685362 32136 catalog_manager.cc:4841] TS 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691): tablet 089e48bdf75a49d0b597b9ec60cdacf4 (table test-table [id=42b4ed38ab0c4976bac4781a6efa234e]) successfully deleted
I20260928 06:31:29.685442 32135 catalog_manager.cc:4841] TS bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659): tablet 089e48bdf75a49d0b597b9ec60cdacf4 (table test-table [id=42b4ed38ab0c4976bac4781a6efa234e]) successfully deleted
I20260928 06:31:29.686197 32662 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: Bootstrap starting.
I20260928 06:31:29.686537 32665 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Bootstrap starting.
I20260928 06:31:29.686564 32661 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Bootstrap starting.
I20260928 06:31:29.686616 32662 tablet_bootstrap.cc:654] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.686866 32665 tablet_bootstrap.cc:654] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.686866 32661 tablet_bootstrap.cc:654] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:29.687450 32661 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: No bootstrap required, opened a new log
I20260928 06:31:29.687505 32661 ts_tablet_manager.cc:1397] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:29.687585 32661 raft_consensus.cc:348] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.687636 32661 raft_consensus.cc:374] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.687649 32661 raft_consensus.cc:729] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1d60d2d5eb4d48b1f677393267cf49, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.687708 32661 consensus_queue.cc:260] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.687808 32661 ts_tablet_manager.cc:1428] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.688261 32662 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: No bootstrap required, opened a new log
I20260928 06:31:29.688326 32662 ts_tablet_manager.cc:1397] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:29.688483 32662 raft_consensus.cc:348] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.688560 32662 raft_consensus.cc:374] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.688585 32662 raft_consensus.cc:729] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eebea79e0934c63985762b0a8926085, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.688637 32662 consensus_queue.cc:260] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.688747 32662 ts_tablet_manager.cc:1428] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:29.689077 32665 tablet_bootstrap.cc:492] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: No bootstrap required, opened a new log
I20260928 06:31:29.689126 32665 ts_tablet_manager.cc:1397] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:31:29.689240 32665 raft_consensus.cc:348] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.689316 32665 raft_consensus.cc:374] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:29.689335 32665 raft_consensus.cc:729] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Initialized, Role: FOLLOWER
I20260928 06:31:29.689383 32665 consensus_queue.cc:260] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.689489 32665 ts_tablet_manager.cc:1428] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:29.762951 32693 raft_consensus.cc:482] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:29.763088 32693 raft_consensus.cc:504] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.763259 32693 leader_election.cc:288] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.763464 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad07e9ccfae54f13a96a5acb76f395e5" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085" is_pre_election: true
I20260928 06:31:29.763571 32306 raft_consensus.cc:2454] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 0.
I20260928 06:31:29.763660 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad07e9ccfae54f13a96a5acb76f395e5" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" is_pre_election: true
I20260928 06:31:29.763747 32380 leader_election.cc:302] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eebea79e0934c63985762b0a8926085, 7b354b1abf3448eb81df079bd3c24d55; no voters:
I20260928 06:31:29.763772 32587 raft_consensus.cc:2454] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 0.
I20260928 06:31:29.763835 32693 raft_consensus.cc:2781] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:29.763883 32693 raft_consensus.cc:482] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:29.763901 32693 raft_consensus.cc:3037] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.764645 32693 raft_consensus.cc:504] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.764820 32693 leader_election.cc:288] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 election: Requested vote from peers 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691), bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659)
I20260928 06:31:29.765010 32306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad07e9ccfae54f13a96a5acb76f395e5" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eebea79e0934c63985762b0a8926085"
I20260928 06:31:29.765028 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad07e9ccfae54f13a96a5acb76f395e5" candidate_uuid: "7b354b1abf3448eb81df079bd3c24d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf1d60d2d5eb4d48b1f677393267cf49"
I20260928 06:31:29.765105 32587 raft_consensus.cc:3037] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.765105 32306 raft_consensus.cc:3037] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:29.765923 32306 raft_consensus.cc:2454] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 1.
I20260928 06:31:29.765933 32587 raft_consensus.cc:2454] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b354b1abf3448eb81df079bd3c24d55 in term 1.
I20260928 06:31:29.766106 32379 leader_election.cc:302] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b354b1abf3448eb81df079bd3c24d55, bf1d60d2d5eb4d48b1f677393267cf49; no voters:
I20260928 06:31:29.766208 32693 raft_consensus.cc:2781] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:29.766258 32693 raft_consensus.cc:686] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 LEADER]: Becoming Leader. State: Replica: 7b354b1abf3448eb81df079bd3c24d55, State: Running, Role: LEADER
I20260928 06:31:29.766302 32693 consensus_queue.cc:237] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } }
I20260928 06:31:29.766894 32147 catalog_manager.cc:5495] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "7b354b1abf3448eb81df079bd3c24d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b354b1abf3448eb81df079bd3c24d55" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:29.783993 32659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.781790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.24.11.130:39181)
0928 06:31:29.781839 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:29.781842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:29.781920 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:31:29.782006 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:29.782007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:29.782015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:29.782068 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:29.782155 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.782621 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.783359 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:29.783558 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:29.783623 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:29.783750 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:29.783776 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:29.783810 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:29.783862 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:31:29.783881 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:29.786324 32587 raft_consensus.cc:1261] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Refusing update from remote peer 7b354b1abf3448eb81df079bd3c24d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.786334 32306 raft_consensus.cc:1261] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Refusing update from remote peer 7b354b1abf3448eb81df079bd3c24d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:29.786523 32669 consensus_queue.cc:1035] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf1d60d2d5eb4d48b1f677393267cf49" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 44659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.786620 32693 consensus_queue.cc:1035] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eebea79e0934c63985762b0a8926085" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:29.797214 32150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.797320 32150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:29.797993 32150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 451d0373a1164de8a9561e0c8b01f718: Sending DeleteTablet for 3 replicas of tablet ad07e9ccfae54f13a96a5acb76f395e5
I20260928 06:31:29.798229 32287 tablet_service.cc:1514] Processing DeleteTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:39874
I20260928 06:31:29.798247 32427 tablet_service.cc:1514] Processing DeleteTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35968
I20260928 06:31:29.798257 32567 tablet_service.cc:1514] Processing DeleteTablet for tablet ad07e9ccfae54f13a96a5acb76f395e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:29 UTC) from {username='oryx', principal='oryx/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56398
I20260928 06:31:29.798338 32685 tablet_replica.cc:330] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: stopping tablet replica
I20260928 06:31:29.798372 32686 tablet_replica.cc:330] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: stopping tablet replica
I20260928 06:31:29.798390 32685 raft_consensus.cc:2229] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:29.798424 32686 raft_consensus.cc:2229] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.798468 32686 raft_consensus.cc:2258] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.798476 32685 raft_consensus.cc:2258] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.798557 32684 tablet_replica.cc:330] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085: stopping tablet replica
I20260928 06:31:29.798631 32684 raft_consensus.cc:2229] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:29.798671 32684 raft_consensus.cc:2258] T ad07e9ccfae54f13a96a5acb76f395e5 P 0eebea79e0934c63985762b0a8926085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:29.799093 32686 ts_tablet_manager.cc:1905] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.799150 32685 ts_tablet_manager.cc:1905] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:29.799412 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 32222
I20260928 06:31:29.800966 32686 ts_tablet_manager.cc:1918] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:29.800971 32685 ts_tablet_manager.cc:1918] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:31:29.801054 32686 log.cc:1196] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/ad07e9ccfae54f13a96a5acb76f395e5
I20260928 06:31:29.801052 32685 log.cc:1196] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/ad07e9ccfae54f13a96a5acb76f395e5
I20260928 06:31:29.801287 32686 ts_tablet_manager.cc:1939] T ad07e9ccfae54f13a96a5acb76f395e5 P bf1d60d2d5eb4d48b1f677393267cf49: Deleting consensus metadata
I20260928 06:31:29.801288 32685 ts_tablet_manager.cc:1939] T ad07e9ccfae54f13a96a5acb76f395e5 P 7b354b1abf3448eb81df079bd3c24d55: Deleting consensus metadata
I20260928 06:31:29.801672 32135 catalog_manager.cc:4841] TS bf1d60d2d5eb4d48b1f677393267cf49 (127.24.11.131:44659): tablet ad07e9ccfae54f13a96a5acb76f395e5 (table test-table [id=df191cd9f3a341488849a95bed09ea3d]) successfully deleted
I20260928 06:31:29.802418 32136 catalog_manager.cc:4841] TS 7b354b1abf3448eb81df079bd3c24d55 (127.24.11.130:39181): tablet ad07e9ccfae54f13a96a5acb76f395e5 (table test-table [id=df191cd9f3a341488849a95bed09ea3d]) successfully deleted
W20260928 06:31:29.805721 32136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:38691 (error 108)
I20260928 06:31:29.805980 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 32362
I20260928 06:31:29.806675 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:29.806151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:38691 (local address 127.0.0.1:39890)
0928 06:31:29.806578 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:29.806624 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:38691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":39,"threads_started":1}
W20260928 06:31:29.806769 32136 catalog_manager.cc:4584] TS 0eebea79e0934c63985762b0a8926085 (127.24.11.129:38691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad07e9ccfae54f13a96a5acb76f395e5: Network error: Client connection negotiation failed: client connection to 127.24.11.129:38691: connect: Connection refused (error 111)
I20260928 06:31:29.812291 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 32502
I20260928 06:31:29.818719 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 32118
2026-09-28T06:31:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2623 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32748](info): set up 2 sockets
Sep 28 06:31:29 dist-test-slave-38pg krb5kdc[32748](info): commencing operation
krb5kdc: starting...
W20260928 06:31:30.859882 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:30 dist-test-slave-38pg krb5kdc[32748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:30.980760 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35765
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:38047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:31.063563 32761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:31.063824 32761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:31.063875 32761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:31.065186 32761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:31.065248 32761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:31.065270 32761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:31.065299 32761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:31.065330 32761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:31.066864 32761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:31.067152 32761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:31.067407 32761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:31.072522 32766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:31.072528 32767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:31.072544 301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:31.072945 32761 server_base.cc:1034] running on GCE node
I20260928 06:31:31.073251 32761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:31.073598 32761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:31.074785 32761 hybrid_clock.cc:648] HybridClock initialized: now 1790577091074778 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:31:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1259 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:31 dist-test-slave-38pg krb5kdc[306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:31 dist-test-slave-38pg krb5kdc[306](info): set up 2 sockets
Sep 28 06:31:31 dist-test-slave-38pg krb5kdc[306](info): commencing operation
krb5kdc: starting...
W20260928 06:31:32.111284 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:32.228499 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41567
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:41567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:32.310465 319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:32.310722 319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:32.310750 319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:32.311954 319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:32.311995 319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:32.312011 319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:32.312023 319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:32.312036 319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:32.313711 319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:41567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:32.313948 319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:32.314155 319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:32.319432 324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.319458 325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.319577 319 server_base.cc:1034] running on GCE node
W20260928 06:31:32.319634 327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.319860 319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:32.320205 319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:32.321378 319 hybrid_clock.cc:648] HybridClock initialized: now 1790577092321349 us; error 60 us; skew 500 ppm
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:32.324071 319 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:32.324564 319 webserver.cc:460] Webserver started at http://127.24.11.190:35951/ using document root <none> and password file <none>
I20260928 06:31:32.324805 319 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:32.324855 319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:32.324960 319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:32.326215 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "1f184f541d63436f8a825b9d5bc4e336"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "e6c465ba01baf6c88a624418949b2f9d"
server_key_iv: "277dd79687aad1064d598e888a12b9b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.326611 319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "1f184f541d63436f8a825b9d5bc4e336"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "e6c465ba01baf6c88a624418949b2f9d"
server_key_iv: "277dd79687aad1064d598e888a12b9b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.328097 319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:32.329077 334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.329289 319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:32.329387 319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "1f184f541d63436f8a825b9d5bc4e336"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "e6c465ba01baf6c88a624418949b2f9d"
server_key_iv: "277dd79687aad1064d598e888a12b9b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.329458 319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:32.348244 319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:32.351277 319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:32.351416 319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:32.355537 319 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:41567
I20260928 06:31:32.355621 386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:41567 every 8 connection(s)
I20260928 06:31:32.356003 319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:32.357102 387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:32.359625 387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Bootstrap starting.
I20260928 06:31:32.360280 387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:32.360592 387 log.cc:824] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:32.361477 387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: No bootstrap required, opened a new log
I20260928 06:31:32.363283 387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:32.363479 387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:32.363528 387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f184f541d63436f8a825b9d5bc4e336, State: Initialized, Role: FOLLOWER
I20260928 06:31:32.363727 387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:32.363830 387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:32.363875 387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:32.363883 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 319
I20260928 06:31:32.363929 387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:32.364048 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:32.364245 24622 external_mini_cluster.cc:1428] Setting key ccee4f902b90dce2a0486e32beb105b7
I20260928 06:31:32.364692 387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:32.364806 387 leader_election.cc:302] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f184f541d63436f8a825b9d5bc4e336; no voters:
I20260928 06:31:32.364989 387 leader_election.cc:288] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:32.365245 392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:32.365267 387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:32.365407 392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 LEADER]: Becoming Leader. State: Replica: 1f184f541d63436f8a825b9d5bc4e336, State: Running, Role: LEADER
I20260928 06:31:32.365507 392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:32.365909 392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f184f541d63436f8a825b9d5bc4e336. Latest consensus state: current_term: 1 leader_uuid: "1f184f541d63436f8a825b9d5bc4e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } } }
I20260928 06:31:32.365919 393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f184f541d63436f8a825b9d5bc4e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } } }
I20260928 06:31:32.366017 393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:32.366163 392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:32.366272 401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:32.366861 401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:32.368637 401 catalog_manager.cc:1347] Generated new cluster ID: 34a7fc17a7b44f74b793567be5897508
I20260928 06:31:32.368703 401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:32.376303 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.364814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.24.11.190:41567)
0928 06:31:32.365025 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:32.365108 (+ 83us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:32.365151 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:31:32.365277 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:32.365285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:32.365399 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:32.366274 (+ 875us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:32.366546 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.368039 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.369270 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.369913 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.370052 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.372596 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:32.372605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:32.372605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:32.372622 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:32.374587 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.375023 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.375029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.375029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.375138 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.375360 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.375362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.375362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.375648 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:32.375822 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:32.375934 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 06:31:32.376093 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260928 06:31:32.388230 401 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:32.389027 401 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:32.395823 401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Generated new TSK 0
I20260928 06:31:32.396064 401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:32.433337 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:32.519059 413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:32.519273 413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:32.519304 413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:32.520455 413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:32.520490 413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:32.520517 413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:32.521960 413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:32.522149 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:32.522336 413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:32.527524 413 server_base.cc:1034] running on GCE node
W20260928 06:31:32.527570 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.527487 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.527482 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.527860 413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:32.528151 413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:32.529333 413 hybrid_clock.cc:648] HybridClock initialized: now 1790577092529318 us; error 26 us; skew 500 ppm
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:32.532027 413 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:32.532598 413 webserver.cc:460] Webserver started at http://127.24.11.129:35781/ using document root <none> and password file <none>
I20260928 06:31:32.532858 413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:32.532932 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:32.533087 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:32.534300 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "c2929cd1af0e41a6bffb1be368bb3878"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06a0c54604298d51deb69a326b2830bb"
server_key_iv: "d10878c66dc7114a2c65ea23e5fbaaa3"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.534682 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "c2929cd1af0e41a6bffb1be368bb3878"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06a0c54604298d51deb69a326b2830bb"
server_key_iv: "d10878c66dc7114a2c65ea23e5fbaaa3"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.536139 413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:32.537003 428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.537262 413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:32.537384 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "c2929cd1af0e41a6bffb1be368bb3878"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06a0c54604298d51deb69a326b2830bb"
server_key_iv: "d10878c66dc7114a2c65ea23e5fbaaa3"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.537467 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:32.551713 413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:32.554473 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:32.554661 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:32.554942 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:32.555280 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:32.555344 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.555382 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:32.555430 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.561532 413 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:44337
I20260928 06:31:32.561594 541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:44337 every 8 connection(s)
I20260928 06:31:32.561930 413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:32.568857 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 413
I20260928 06:31:32.568984 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:32.569171 24622 external_mini_cluster.cc:1428] Setting key 2c8aef6c2e03a77bf49cb01841021a91
I20260928 06:31:32.572870 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.562661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:34899)
0928 06:31:32.562864 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.562907 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:32.563399 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:32.563608 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:32.563611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:32.564188 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:32.564399 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.564404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.565103 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.565104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.566067 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.566070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.566469 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.566470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.566575 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.567169 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:32.567177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:32.568664 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:32.570817 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.570823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.570826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.571142 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.571437 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.571438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.571438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.571595 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.572025 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:32.572032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:32.572206 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 06:31:32.572375 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 06:31:32.572598 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
I20260928 06:31:32.573763 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.562673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:34899 (local address 127.24.11.190:41567)
0928 06:31:32.562746 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:32.562749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:32.562980 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:31:32.563484 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:32.563485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:32.563496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:32.563551 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:32.564481 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.565014 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.566140 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.566398 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.566454 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.568773 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:32.568783 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:32.568783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:32.568793 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:32.570708 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.571242 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.571244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.571244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.571326 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.571667 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.571668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.571669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.571841 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:32.572002 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:32.572418 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 06:31:32.573644 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:31:32.575278 542 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41567
I20260928 06:31:32.575388 542 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:32.575605 542 heartbeater.cc:511] Master 127.24.11.190:41567 requested a full tablet report, sending...
I20260928 06:31:32.576373 351 ts_manager.cc:194] Registered new tserver with Master: c2929cd1af0e41a6bffb1be368bb3878 (127.24.11.129:44337)
I20260928 06:31:32.577380 351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:34899
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:32.597133 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:32.681484 547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:32.681840 547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:32.681882 547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:32.683147 547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:32.683228 547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:32.683274 547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:32.684989 547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:32.685303 547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:32.685631 547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:32.690884 552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.690884 553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.690860 555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.691157 547 server_base.cc:1034] running on GCE node
I20260928 06:31:32.691390 547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:32.691650 547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:32.692864 547 hybrid_clock.cc:648] HybridClock initialized: now 1790577092692845 us; error 42 us; skew 500 ppm
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:32.695662 547 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:32.696180 547 webserver.cc:460] Webserver started at http://127.24.11.130:40221/ using document root <none> and password file <none>
I20260928 06:31:32.696444 547 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:32.696516 547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:32.696668 547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:32.697839 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "9ea7568a7e92456d9ef874fed255e3d1"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06382273bf478e8e394950ac462569b2"
server_key_iv: "3bdd461e2fd7e819feaceb6c7978dcf2"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.698202 547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "9ea7568a7e92456d9ef874fed255e3d1"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06382273bf478e8e394950ac462569b2"
server_key_iv: "3bdd461e2fd7e819feaceb6c7978dcf2"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.699666 547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:32.700466 562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.700634 547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:32.700743 547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "9ea7568a7e92456d9ef874fed255e3d1"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06382273bf478e8e394950ac462569b2"
server_key_iv: "3bdd461e2fd7e819feaceb6c7978dcf2"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.700867 547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:32.726920 547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:32.729439 547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:32.729655 547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:32.729916 547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:32.730244 547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:32.730306 547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.730345 547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:32.730396 547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.736642 547 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37531
I20260928 06:31:32.736727 675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37531 every 8 connection(s)
I20260928 06:31:32.737079 547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:32.742687 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 547
I20260928 06:31:32.742828 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:32.743043 24622 external_mini_cluster.cc:1428] Setting key 2c120859956da4a413637a866c0f4398
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:32.748311 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.738067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:53715 (local address 127.24.11.190:41567)
0928 06:31:32.738154 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:32.738156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:32.738270 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:31:32.738757 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:32.738759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:32.738769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:32.738836 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:32.739872 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.740488 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.741766 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.741989 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.742064 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.744852 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:32.744854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:32.744855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:32.744864 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:32.746640 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.747196 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.747197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.747198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.747260 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.747595 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.747597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.747597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.747734 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:32.747793 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:32.748153 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 06:31:32.748228 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:31:32.748638 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.737959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:53715)
0928 06:31:32.738137 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.738177 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:32.738661 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:32.738892 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:32.738895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:32.739501 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:32.739704 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.739710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.740603 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.740604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.741674 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.741677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.742099 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.742101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.742216 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.743009 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:32.743021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:32.744750 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:32.746741 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.746748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.746751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.747098 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.747352 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.747353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.747353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.747510 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.747806 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:32.747813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:32.747998 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 06:31:32.748127 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:31:32.748331 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 06:31:32.749233 676 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41567
I20260928 06:31:32.749356 676 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:32.749586 676 heartbeater.cc:511] Master 127.24.11.190:41567 requested a full tablet report, sending...
I20260928 06:31:32.750017 351 ts_manager.cc:194] Registered new tserver with Master: 9ea7568a7e92456d9ef874fed255e3d1 (127.24.11.130:37531)
I20260928 06:31:32.750587 351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:53715
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:32.773993 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:32.862648 681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:32.862871 681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:32.862898 681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:32.864212 681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:32.864259 681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:32.864290 681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:32.866124 681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:32.866432 681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:32.866711 681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:32.872262 687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:32.872246 686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.872515 681 server_base.cc:1034] running on GCE node
W20260928 06:31:32.872238 689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:32.872853 681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:32.873104 681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:32.874279 681 hybrid_clock.cc:648] HybridClock initialized: now 1790577092874249 us; error 44 us; skew 500 ppm
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:32.877097 681 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:32.877664 681 webserver.cc:460] Webserver started at http://127.24.11.131:44177/ using document root <none> and password file <none>
I20260928 06:31:32.877924 681 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:32.877972 681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:32.878121 681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:32.879220 681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "f39ac69b83d24e1dba682f05c3226066"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "30b57622a70f2408ed177e2d381a5f9c"
server_key_iv: "b1b4ae9f07396d8f09bab50668c5249e"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.879577 681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "f39ac69b83d24e1dba682f05c3226066"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "30b57622a70f2408ed177e2d381a5f9c"
server_key_iv: "b1b4ae9f07396d8f09bab50668c5249e"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.881022 681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:32.881999 696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.882220 681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:32.882311 681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "f39ac69b83d24e1dba682f05c3226066"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "30b57622a70f2408ed177e2d381a5f9c"
server_key_iv: "b1b4ae9f07396d8f09bab50668c5249e"
server_key_version: "encryptionkey@0"
I20260928 06:31:32.882380 681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:32.905287 681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:32.907423 681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:32.907552 681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:32.907752 681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:32.908110 681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:32.908149 681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.908198 681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:32.908228 681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:32.914542 681 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:33967
I20260928 06:31:32.914584 809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:33967 every 8 connection(s)
I20260928 06:31:32.914987 681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:32.919634 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 681
I20260928 06:31:32.919760 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:32.919984 24622 external_mini_cluster.cc:1428] Setting key 1a9f5c088d250e22c73d5407123075b6
Sep 28 06:31:32 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577092, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:32.924705 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.915775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42019 (local address 127.24.11.190:41567)
0928 06:31:32.915846 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:32.915848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:32.916090 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:31:32.916559 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:32.916560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:32.916571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:32.916645 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:32.917640 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.918185 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.919183 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.919475 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.919546 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.922183 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:32.922184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:32.922185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:32.922192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:32.923421 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.923805 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.923806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.923807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.923869 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:32.924147 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:32.924148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:32.924148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:32.924253 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:32.924319 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:32.924572 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:31:32.924642 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:31:32.924785 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:32.915756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:42019)
0928 06:31:32.915949 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:32.915992 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:32.916472 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:32.916706 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:32.916709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:32.917270 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:32.917494 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.917499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.918266 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.918267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.919103 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:32.919105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:32.919579 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:32.919581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:32.919689 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:32.920376 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:32.920386 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:32.922091 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:32.923493 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.923499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.923501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.923717 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.923946 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:32.923947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:32.923947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:32.924084 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:32.924311 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:32.924314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:32.924431 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:31:32.924510 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:31:32.924628 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260928 06:31:32.925326 810 heartbeater.cc:347] Connected to a master server at 127.24.11.190:41567
I20260928 06:31:32.925436 810 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:32.925630 810 heartbeater.cc:511] Master 127.24.11.190:41567 requested a full tablet report, sending...
I20260928 06:31:32.926038 351 ts_manager.cc:194] Registered new tserver with Master: f39ac69b83d24e1dba682f05c3226066 (127.24.11.131:33967)
I20260928 06:31:32.926369 351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:42019
I20260928 06:31:32.931764 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
Entry for principal oryx/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab.
WARNING: no policy specified for oryx/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
Entry for principal oryx/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab.
WARNING: no policy specified for oryx/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
Entry for principal oryx/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab.
I20260928 06:31:33.005228 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 413
I20260928 06:31:33.009812 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 547
I20260928 06:31:33.014485 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 681
I20260928 06:31:33.019048 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 319
I20260928 06:31:33.024327 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41567
--webserver_interface=127.24.11.190
--webserver_port=35951
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:41567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:33.107553 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:33.107810 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:33.107848 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:33.109053 819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:33.109125 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:33.109149 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:33.109169 819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:33.109202 819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:33.110741 819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:41567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=35951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:33.111043 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:33.111284 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:31:33.116321 819 server_base.cc:1034] running on GCE node
W20260928 06:31:33.116288 827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.116302 825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.116287 824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:33.116740 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:33.117050 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:33.118217 819 hybrid_clock.cc:648] HybridClock initialized: now 1790577093118193 us; error 37 us; skew 500 ppm
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577093, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:33.121249 819 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:33.121783 819 webserver.cc:460] Webserver started at http://127.24.11.190:35951/ using document root <none> and password file <none>
I20260928 06:31:33.122046 819 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:33.122107 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:33.123484 819 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:33.124191 834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.124401 819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:33.124496 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "1f184f541d63436f8a825b9d5bc4e336"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "e6c465ba01baf6c88a624418949b2f9d"
server_key_iv: "277dd79687aad1064d598e888a12b9b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:33.124852 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:33.137102 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:33.139313 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:33.139477 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:33.143458 819 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:41567
I20260928 06:31:33.143491 886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:41567 every 8 connection(s)
I20260928 06:31:33.143867 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:33.145315 887 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:31:33.146030 887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Bootstrap starting.
I20260928 06:31:33.149183 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 819
I20260928 06:31:33.149513 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:44337
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:41567
--webserver_port=35781
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:31:33.155237 887 log.cc:824] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:33.157774 887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:31:33.158129 887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Bootstrap complete.
I20260928 06:31:33.161926 887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:33.162245 887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f184f541d63436f8a825b9d5bc4e336, State: Initialized, Role: FOLLOWER
I20260928 06:31:33.162420 887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:33.162487 887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:33.162523 887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:33.162568 887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:31:33.163897 887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:33.164012 887 leader_election.cc:302] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f184f541d63436f8a825b9d5bc4e336; no voters:
I20260928 06:31:33.164201 887 leader_election.cc:288] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:31:33.164328 891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:31:33.164453 887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:33.164490 891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [term 2 LEADER]: Becoming Leader. State: Replica: 1f184f541d63436f8a825b9d5bc4e336, State: Running, Role: LEADER
I20260928 06:31:33.164604 891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } }
I20260928 06:31:33.164865 892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1f184f541d63436f8a825b9d5bc4e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } } }
I20260928 06:31:33.164942 892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:33.165194 893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f184f541d63436f8a825b9d5bc4e336. Latest consensus state: current_term: 2 leader_uuid: "1f184f541d63436f8a825b9d5bc4e336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f184f541d63436f8a825b9d5bc4e336" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 41567 } } }
I20260928 06:31:33.165319 893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:33.165447 899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:33.165990 899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:33.166123 899 catalog_manager.cc:1259] Loaded cluster ID: 34a7fc17a7b44f74b793567be5897508
I20260928 06:31:33.166338 899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:33.167490 899 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:33.167668 899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1f184f541d63436f8a825b9d5bc4e336: Loaded TSK: 0
I20260928 06:31:33.167882 899 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:31:33.237860 889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:33.238121 889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:33.238166 889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:33.239434 889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:33.239506 889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:33.239554 889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:33.241113 889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.129.keytab
--principal=oryx/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:44337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=35781
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:33.241397 889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:33.241701 889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:33.248441 914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.248443 913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.248479 916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:33.248764 889 server_base.cc:1034] running on GCE node
I20260928 06:31:33.249083 889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:33.249356 889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:33.250527 889 hybrid_clock.cc:648] HybridClock initialized: now 1790577093250511 us; error 36 us; skew 500 ppm
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:33.253336 889 init.cc:377] Logged in from keytab as oryx/127.24.11.129@KRBTEST.COM (short username oryx)
I20260928 06:31:33.253916 889 webserver.cc:460] Webserver started at http://127.24.11.129:35781/ using document root <none> and password file <none>
I20260928 06:31:33.254172 889 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:33.254256 889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:33.255739 889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:33.256409 923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.256624 889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:33.256737 889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "c2929cd1af0e41a6bffb1be368bb3878"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06a0c54604298d51deb69a326b2830bb"
server_key_iv: "d10878c66dc7114a2c65ea23e5fbaaa3"
server_key_version: "encryptionkey@0"
I20260928 06:31:33.257122 889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:33.276309 889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:33.278769 889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:33.278918 889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:33.279189 889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:33.279520 889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:33.279574 889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.279608 889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:33.279636 889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.285953 889 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:44337
I20260928 06:31:33.286023 1036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:44337 every 8 connection(s)
I20260928 06:31:33.286377 889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:33.295852 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 889
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.296177 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:37531
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:41567
--webserver_port=40221
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.297996 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.287153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:38643)
0928 06:31:33.287363 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.287409 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.288023 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.290213 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.290217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.290806 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.291019 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.291024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.292720 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.292721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.293502 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.293504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.294445 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.294446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.294547 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.295190 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.295199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.297686 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260928 06:31:33.298015 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.287280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:38643 (local address 127.24.11.190:41567)
0928 06:31:33.289712 (+ 2432us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.289728 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.289787 (+ 59us) server_negotiation.cc:407] Connection header received
0928 06:31:33.289934 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.289942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.290006 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.290154 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.291153 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.292619 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.293660 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.294333 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.294455 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.297842 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:38643: BlockingRecv error: recv got EOF from 127.24.11.129:38643 (error 108)
Metrics: {"server-negotiator.queue_time_us":2363,"thread_start_us":74,"threads_started":1}
W20260928 06:31:33.298221 1037 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:41567 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.11.190:41567: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.303932 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.298464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:56531)
0928 06:31:33.298509 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.298521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.298558 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.298685 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.298686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.298869 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.298923 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.298925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.299837 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.299838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.300748 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.300751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.301212 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.301213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.301289 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.301775 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.301781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.303868 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:31:33.304071 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.298510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:56531 (local address 127.24.11.190:41567)
0928 06:31:33.298541 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.298543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.298552 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:33.298609 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.298610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.298621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.298679 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.298994 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.299730 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.300831 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.301133 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.301198 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.303986 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:56531: BlockingRecv error: recv got EOF from 127.24.11.129:56531 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.309231 1039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.304465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:56753)
0928 06:31:33.304500 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.304509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.304551 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.304717 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.304718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.304861 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.304895 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.304896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.305921 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.305922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.306399 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.306400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.306894 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.306895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.306933 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.307392 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.307397 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.309160 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:31:33.309360 1037 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:31:33.309376 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.304538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:56753 (local address 127.24.11.190:41567)
0928 06:31:33.304578 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.304580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.304609 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:31:33.304632 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.304633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.304644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.304704 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.304978 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.305821 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.306485 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.306810 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.306892 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.309237 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:56753: BlockingRecv error: recv got EOF from 127.24.11.129:56753 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:31:33.379978 1041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:33.380196 1041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:33.380220 1041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:33.381436 1041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:33.381481 1041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:33.381508 1041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:33.383029 1041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.130.keytab
--principal=oryx/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:37531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=40221
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:33.383272 1041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:33.383513 1041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:33.388625 1046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.388648 1049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.388638 1047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:33.389020 1041 server_base.cc:1034] running on GCE node
I20260928 06:31:33.389276 1041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:33.389606 1041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:33.390758 1041 hybrid_clock.cc:648] HybridClock initialized: now 1790577093390720 us; error 55 us; skew 500 ppm
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:33.393392 1041 init.cc:377] Logged in from keytab as oryx/127.24.11.130@KRBTEST.COM (short username oryx)
I20260928 06:31:33.393925 1041 webserver.cc:460] Webserver started at http://127.24.11.130:40221/ using document root <none> and password file <none>
I20260928 06:31:33.394160 1041 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:33.394239 1041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:33.395615 1041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:31:33.396239 1056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.396425 1041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:31:33.396523 1041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "9ea7568a7e92456d9ef874fed255e3d1"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "06382273bf478e8e394950ac462569b2"
server_key_iv: "3bdd461e2fd7e819feaceb6c7978dcf2"
server_key_version: "encryptionkey@0"
I20260928 06:31:33.396891 1041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:33.435566 1041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:33.438122 1041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:33.438297 1041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:33.438557 1041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:33.438890 1041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:33.438946 1041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.438992 1041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:33.439018 1041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.444897 1041 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37531
I20260928 06:31:33.444942 1171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37531 every 8 connection(s)
I20260928 06:31:33.445312 1041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.452275 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1041
I20260928 06:31:33.452608 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:33967
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:41567
--webserver_port=44177
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:33.453073 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.446111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:57057 (local address 127.24.11.190:41567)
0928 06:31:33.446153 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.446155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.446395 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:31:33.446940 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.446941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.446959 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.447024 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.448005 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.448503 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.449605 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.449802 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.449896 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.453023 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:57057: BlockingRecv error: recv got EOF from 127.24.11.130:57057 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:31:33.453181 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.446100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:57057)
0928 06:31:33.446262 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.446299 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.446827 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.447098 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.447102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.447694 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.447896 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.447900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.448602 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.448603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.449463 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.449466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.449908 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.449909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.450036 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.450577 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.450586 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.452933 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
W20260928 06:31:33.453430 1172 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:41567 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.11.190:41567: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.458521 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.453656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:50159)
0928 06:31:33.453806 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.453817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.453868 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.454048 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.454049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.454198 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.454241 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.454243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.455151 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.455152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.455651 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.455653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.456034 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.456035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.456103 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.456646 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.456650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.458413 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
W20260928 06:31:33.458542 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.453779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:50159 (local address 127.24.11.190:41567)
0928 06:31:33.453821 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.453823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.453854 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:31:33.453942 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.453943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.453954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.454030 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.454313 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.455074 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.455753 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.455948 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.456033 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.458497 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:50159: BlockingRecv error: recv got EOF from 127.24.11.130:50159 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.463632 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.458803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:34385)
0928 06:31:33.458880 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.458894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.458954 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.459084 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.459085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.459245 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.459287 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.459289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.459872 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.459873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.460783 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.460786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.461216 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.461217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.461284 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.461729 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.461735 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.463546 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 06:31:33.463682 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.458872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:34385 (local address 127.24.11.190:41567)
0928 06:31:33.458935 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.458936 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.458945 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:33.459019 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.459020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.459028 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.459075 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.459361 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.459801 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.460922 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.461149 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.461199 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.463619 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:34385: BlockingRecv error: recv got EOF from 127.24.11.130:34385 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:31:33.464010 1172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:31:33.536089 1176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:33.536309 1176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:33.536334 1176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:33.537580 1176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:33.537622 1176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:33.537652 1176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:33.539151 1176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/krb5kdc/oryx_127.24.11.131.keytab
--principal=oryx/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:33967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=44177
--tserver_master_addrs=127.24.11.190:41567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:33.539409 1176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:33.539655 1176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:33.544581 1184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.544581 1182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:33.544587 1181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:33.544823 1176 server_base.cc:1034] running on GCE node
I20260928 06:31:33.545061 1176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:33.545267 1176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:33.546428 1176 hybrid_clock.cc:648] HybridClock initialized: now 1790577093546414 us; error 28 us; skew 500 ppm
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:33.555174 1176 init.cc:377] Logged in from keytab as oryx/127.24.11.131@KRBTEST.COM (short username oryx)
I20260928 06:31:33.555697 1176 webserver.cc:460] Webserver started at http://127.24.11.131:44177/ using document root <none> and password file <none>
I20260928 06:31:33.555977 1176 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:33.556071 1176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:33.557472 1176 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:31:33.558132 1191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.558301 1176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.558406 1176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "f39ac69b83d24e1dba682f05c3226066"
format_stamp: "Formatted at 2026-09-28 06:31:32 on dist-test-slave-38pg"
server_key: "30b57622a70f2408ed177e2d381a5f9c"
server_key_iv: "b1b4ae9f07396d8f09bab50668c5249e"
server_key_version: "encryptionkey@0"
I20260928 06:31:33.558785 1176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:33.575136 1176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:33.577709 1176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:33.577888 1176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:33.578155 1176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:33.578480 1176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:33.578538 1176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.578580 1176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:33.578608 1176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:33.584506 1176 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:33967
I20260928 06:31:33.584553 1305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:33967 every 8 connection(s)
I20260928 06:31:33.584936 1176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:33.588227 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1176
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:31:33.592870 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.585721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55259 (local address 127.24.11.190:41567)
0928 06:31:33.585789 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.585791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.586032 (+ 241us) server_negotiation.cc:407] Connection header received
0928 06:31:33.586502 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.586503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.586513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.586577 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.587488 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.587988 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.589425 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.589753 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.589815 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.592824 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:55259: BlockingRecv error: recv got EOF from 127.24.11.131:55259 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:31:33.593003 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.585698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:55259)
0928 06:31:33.585872 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.585914 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.586406 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.586630 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.586633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.587187 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.587381 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.587386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.588092 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.588093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.589334 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.589336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.589830 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.589831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.589982 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.590485 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.590494 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.592726 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260928 06:31:33.593216 1306 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:41567 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.11.190:41567: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
I20260928 06:31:33.596251 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.588748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.24.11.190:41567)
0928 06:31:33.588954 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.588956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.588962 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:31:33.588982 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.588983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.588993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.589051 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.589764 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.590468 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.591234 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.591429 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.591516 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.592807 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:33.592812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:33.592812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:33.592824 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:33.594712 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:33.595140 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:33.595143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:33.595143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:33.595229 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:33.595489 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:33.595490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:33.595491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:33.595717 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:33.595873 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:33.596015 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:31:33.596119 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":42,"threads_started":1}
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.597975 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.593435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:35813)
0928 06:31:33.593515 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.593550 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.593589 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.593704 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.593705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.593831 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.593872 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.593873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.594496 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.594497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.595219 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.595221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.595580 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.595581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.595645 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.596042 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.596047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.597929 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 06:31:33.598140 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.593562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:35813 (local address 127.24.11.190:41567)
0928 06:31:33.593603 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.593605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.593613 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:33.593634 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.593635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.593644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.593700 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.593954 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.594419 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.595297 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.595497 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.595578 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.598077 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:35813: BlockingRecv error: recv got EOF from 127.24.11.131:35813 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:33 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:33.602072 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:33.598261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:46731)
0928 06:31:33.598342 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:33.598352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:33.598393 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:33.598494 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:33.598495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:33.598594 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:33.598619 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.598620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.599174 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.599175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.599694 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:33.599697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.600034 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.600035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:33.600071 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.600408 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:33.600422 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:33.602029 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 06:31:33.602161 1040 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:33.598297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:46731 (local address 127.24.11.190:41567)
0928 06:31:33.598332 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:33.598334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:33.598384 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:31:33.598442 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:33.598443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:33.598451 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:33.598493 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:33.598673 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.599113 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.599783 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:33.599973 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:33.600024 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:33.602102 (+ 2078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:46731: BlockingRecv error: recv got EOF from 127.24.11.131:46731 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:31:33.602236 1306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:34.314682 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.309753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:41309)
0928 06:31:34.309926 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:34.309938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:34.309999 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:34.310162 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:34.310163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:34.310347 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:34.310398 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.310399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.311079 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.311080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.311697 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.311699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.312060 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.312061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.312106 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.312674 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:34.312679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:34.314617 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
W20260928 06:31:34.314855 1312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:34.309885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:41309 (local address 127.24.11.190:41567)
0928 06:31:34.310028 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.310030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.310038 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:34.310061 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.310062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.310071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.310129 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.310481 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.310979 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.311807 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.311994 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.312044 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.314766 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:41309: BlockingRecv error: recv got EOF from 127.24.11.129:41309 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:34.469074 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.464458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:56143)
0928 06:31:34.464590 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:34.464611 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:34.464686 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:34.464818 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:34.464819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:34.465002 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:34.465077 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.465079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.465841 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.465842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.466391 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.466392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.466745 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.466746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.466800 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.467246 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:34.467250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:34.469013 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
W20260928 06:31:34.469185 1312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:34.464539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:56143 (local address 127.24.11.190:41567)
0928 06:31:34.464596 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.464598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.464669 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:31:34.464750 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.464751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.464761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.464812 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.465150 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.465741 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.466494 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.466675 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.466731 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.469110 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:56143: BlockingRecv error: recv got EOF from 127.24.11.130:56143 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:34 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:34.607580 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:34.602692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:54757)
0928 06:31:34.602912 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:34.602926 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:34.603034 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:34.603218 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:34.603219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:34.603419 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:34.603465 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.603467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.604165 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.604166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.604710 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:34.604712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.605091 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.605092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:34.605155 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.605673 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:34.605678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:34.607463 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
W20260928 06:31:34.607618 1312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:34.602842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54757 (local address 127.24.11.190:41567)
0928 06:31:34.602932 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:34.602935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:34.603016 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:31:34.603126 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:34.603126 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:34.603139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:34.603209 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:34.603555 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.604073 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.604810 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:34.605002 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:34.605089 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:34.607549 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:54757: BlockingRecv error: recv got EOF from 127.24.11.131:54757 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:35.319871 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.315175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:60469)
0928 06:31:35.315442 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.315452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.315522 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.315692 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.315693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.315897 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:35.315945 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.315948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.316591 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.316592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.317134 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.317136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.317465 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.317466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.317601 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.318053 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:35.318058 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:35.319814 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":215,"threads_started":1}
W20260928 06:31:35.320029 1315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:35.315209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60469 (local address 127.24.11.190:41567)
0928 06:31:35.315381 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.315383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.315567 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:31:35.315598 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.315599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.315614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.315673 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:35.316019 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.316523 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.317207 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.317397 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.317451 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.319930 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:60469: BlockingRecv error: recv got EOF from 127.24.11.129:60469 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:35.474403 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.469698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:57801)
0928 06:31:35.469833 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.469847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.469922 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.470086 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.470087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.470262 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:35.470322 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.470325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.471005 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.471006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.471534 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.471536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.471936 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.471937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.471997 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.472508 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:35.472512 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:35.474344 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20260928 06:31:35.474542 1315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:35.469810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:57801 (local address 127.24.11.190:41567)
0928 06:31:35.469864 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.469867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.469926 (+ 59us) server_negotiation.cc:407] Connection header received
0928 06:31:35.469993 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.469994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.470007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.470080 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:35.470409 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.470908 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.471620 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.471856 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.471937 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.474460 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:57801: BlockingRecv error: recv got EOF from 127.24.11.130:57801 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:35 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:35.612712 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:35.608186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:47789)
0928 06:31:35.608345 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:35.608356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:35.608405 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:35.608561 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:35.608562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:35.608720 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:35.608769 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.608771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.609370 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.609371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.609926 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:35.609928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.610236 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.610237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:35.610278 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.610819 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:35.610823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:35.612654 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
W20260928 06:31:35.612840 1315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:35.608322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:47789 (local address 127.24.11.190:41567)
0928 06:31:35.608397 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:35.608398 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:35.608407 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:35.608482 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:35.608482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:35.608494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:35.608544 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:35.608827 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.609301 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.609996 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:35.610170 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:35.610220 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:35.612750 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:47789: BlockingRecv error: recv got EOF from 127.24.11.131:47789 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:36.325480 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:36.320503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:57483)
0928 06:31:36.320653 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:36.320665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:36.320718 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:36.320935 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:36.320936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:36.321104 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:36.321153 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.321155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.321859 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.321860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.322389 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.322391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.322736 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.322737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.322778 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.323315 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:36.323321 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:36.325410 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260928 06:31:36.325641 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:36.320589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:57483 (local address 127.24.11.190:41567)
0928 06:31:36.320757 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:36.320759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:36.320768 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:36.320829 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:36.320829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:36.320840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:36.320911 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:36.321227 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.321757 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.322480 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.322669 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.322724 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.325505 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:57483: BlockingRecv error: recv got EOF from 127.24.11.129:57483 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:36.479954 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:36.475000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:38991)
0928 06:31:36.475155 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:36.475170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:36.475241 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:36.475392 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:36.475392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:36.475578 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:36.475626 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.475627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.476314 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.476315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.476860 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.476862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.477271 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.477272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.477340 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.478123 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:36.478130 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:36.479882 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
W20260928 06:31:36.480121 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:36.475142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:38991 (local address 127.24.11.190:41567)
0928 06:31:36.475217 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:36.475219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:36.475231 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:36.475318 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:36.475318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:36.475331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:36.475377 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:36.475718 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.476220 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.476977 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.477179 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.477270 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.480014 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:38991: BlockingRecv error: recv got EOF from 127.24.11.130:38991 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:36 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:36.617980 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:36.613270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:55165)
0928 06:31:36.613471 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:36.613483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:36.613606 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:36.613761 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:36.613762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:36.613952 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:36.613998 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.614000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.614663 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.614665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.615211 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:36.615213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.615588 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.615589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:36.615643 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.616107 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:36.616112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:36.617885 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260928 06:31:36.618067 1320 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:36.613379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55165 (local address 127.24.11.190:41567)
0928 06:31:36.613467 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:36.613469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:36.613571 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:31:36.613663 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:36.613664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:36.613680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:36.613755 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:36.614080 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.614575 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.615306 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:36.615502 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:36.615589 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:36.618003 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:55165: BlockingRecv error: recv got EOF from 127.24.11.131:55165 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:37.331992 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:37.326056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:42447)
0928 06:31:37.326169 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:37.326180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:37.326247 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:37.326453 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:37.326454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:37.326609 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:37.326656 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.326657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.327851 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.327852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.328497 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.328500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.329018 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.329018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.329061 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.329628 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:37.329635 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:37.331932 (+ 2297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260928 06:31:37.332141 1324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:37.326124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:42447 (local address 127.24.11.190:41567)
0928 06:31:37.326263 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:37.326265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:37.326277 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:31:37.326306 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:37.326307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:37.326343 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:37.326450 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:37.326760 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.327714 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.328565 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.328930 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.329021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.332049 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:42447: BlockingRecv error: recv got EOF from 127.24.11.129:42447 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:37.484942 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:37.480451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:33789)
0928 06:31:37.480602 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:37.480621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:37.480690 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:37.480840 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:37.480841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:37.481005 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:37.481052 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.481054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.481701 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.481701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.482221 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.482223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.482554 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.482555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.482595 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.483144 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:37.483149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:37.484883 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
W20260928 06:31:37.485093 1324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:37.480556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33789 (local address 127.24.11.190:41567)
0928 06:31:37.480605 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:37.480607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:37.480673 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:31:37.480744 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:37.480745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:37.480757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:37.480797 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:37.481117 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.481635 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.482300 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.482486 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.482542 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.484999 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:33789: BlockingRecv error: recv got EOF from 127.24.11.130:33789 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:37 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:37.623402 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:37.618499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:55063)
0928 06:31:37.618675 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:37.618687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:37.618775 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:37.618982 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:37.618983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:37.619161 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:37.619206 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.619208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.619870 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.619872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.620394 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:37.620396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.620755 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.620756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:37.620811 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.621314 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:37.621318 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:37.623307 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
W20260928 06:31:37.623441 1324 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:37.618589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55063 (local address 127.24.11.190:41567)
0928 06:31:37.618658 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:37.618660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:37.618753 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:31:37.618855 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:37.618858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:37.618874 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:37.618972 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:37.619286 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.619783 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.620488 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:37.620681 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:37.620759 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:37.623383 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:55063: BlockingRecv error: recv got EOF from 127.24.11.131:55063 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:38.337342 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:38.332724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:58269)
0928 06:31:38.332895 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:38.332909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:38.332983 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:38.333136 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:38.333137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:38.333292 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:38.333343 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.333344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.334069 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.334070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.334585 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.334587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.334941 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.334942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.334984 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.335510 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:38.335515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:38.337282 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
W20260928 06:31:38.337493 1328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:38.332796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58269 (local address 127.24.11.190:41567)
0928 06:31:38.332933 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:38.332934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:38.332957 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:31:38.333043 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:38.333044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:38.333054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:38.333121 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:38.333409 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.334000 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.334654 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.334872 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.334927 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.337402 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:58269: BlockingRecv error: recv got EOF from 127.24.11.129:58269 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:38.490159 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:38.485471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:50183)
0928 06:31:38.485614 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:38.485634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:38.485715 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:38.485879 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:38.485881 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:38.486072 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:38.486118 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.486120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.486794 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.486796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.487322 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.487324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.487690 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.487691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.487750 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.488228 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:38.488232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:38.490098 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":107,"threads_started":1}
W20260928 06:31:38.490290 1328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:38.485624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:50183 (local address 127.24.11.190:41567)
0928 06:31:38.485693 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:38.485696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:38.485706 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:38.485787 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:38.485788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:38.485807 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:38.485873 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:38.486206 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.486701 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.487411 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.487614 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.487692 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.490203 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:50183: BlockingRecv error: recv got EOF from 127.24.11.130:50183 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:38 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:38.634464 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:38.624069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:49921)
0928 06:31:38.624198 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:38.624208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:38.624268 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:38.624418 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:38.624419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:38.624580 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:38.624627 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.624629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.625309 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.625309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.626042 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:38.626044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.626361 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.626361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:38.626403 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.632547 (+ 6144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:38.632555 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:38.634400 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
W20260928 06:31:38.634598 1328 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:38.624156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:49921 (local address 127.24.11.190:41567)
0928 06:31:38.624224 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:38.624226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:38.624255 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:31:38.624337 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:38.624338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:38.624350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:38.624396 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:38.624691 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.625204 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.626110 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:38.626288 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:38.626343 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:38.634516 (+ 8173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:49921: BlockingRecv error: recv got EOF from 127.24.11.131:49921 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:39.342871 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:39.337862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:43165)
0928 06:31:39.338033 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:39.338047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:39.338143 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:39.338343 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:39.338344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:39.338567 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:39.338625 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.338628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.339316 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.339317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.339874 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.339876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.340249 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.340250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.340310 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.340811 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:39.340815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:39.342807 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
W20260928 06:31:39.343050 1332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:39.337874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:43165 (local address 127.24.11.190:41567)
0928 06:31:39.338038 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:39.338041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:39.338122 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:31:39.338237 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:39.338237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:39.338257 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:39.338335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:39.338701 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.339225 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.339972 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.340173 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.340252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.342971 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:43165: BlockingRecv error: recv got EOF from 127.24.11.129:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:39.495860 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:39.490707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:33783)
0928 06:31:39.490932 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:39.490944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:39.491023 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:39.491156 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:39.491157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:39.491365 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:39.491411 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.491413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.492097 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.492099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.492680 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.492681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.493083 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.493084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.493142 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.493678 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:39.493683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:39.495758 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":152,"threads_started":1}
W20260928 06:31:39.496078 1332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:39.490800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33783 (local address 127.24.11.190:41567)
0928 06:31:39.490908 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:39.490910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:39.490996 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:31:39.491083 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:39.491083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:39.491097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:39.491143 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:39.491497 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.492008 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.492769 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.493007 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.493086 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.495943 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:33783: BlockingRecv error: recv got EOF from 127.24.11.130:33783 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:39 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:39.639835 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:39.635069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:53181)
0928 06:31:39.635187 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:39.635205 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:39.635265 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:39.635408 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:39.635409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:39.635568 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:39.635645 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.635648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.636507 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.636508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.637073 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:39.637074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.637441 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.637442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:39.637484 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.637980 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:39.637984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:39.639779 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
W20260928 06:31:39.639992 1332 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:39.635127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:53181 (local address 127.24.11.190:41567)
0928 06:31:39.635193 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:39.635199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:39.635270 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:31:39.635313 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:39.635313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:39.635328 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:39.635406 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:39.635733 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.636435 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.637173 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:39.637368 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:39.637427 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:39.639904 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:53181: BlockingRecv error: recv got EOF from 127.24.11.131:53181 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:40.348357 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:40.343400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:33505)
0928 06:31:40.343624 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:40.343636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:40.343696 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:40.343844 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:40.343845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:40.344026 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:40.344075 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.344077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.344745 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.344746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.345279 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.345280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.345646 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.345647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.345843 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.346333 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:40.346338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:40.348258 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
W20260928 06:31:40.348428 1336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:40.343544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33505 (local address 127.24.11.190:41567)
0928 06:31:40.343684 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:40.343686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:40.343696 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:40.343758 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:40.343758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:40.343769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:40.343819 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:40.344153 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.344660 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.345363 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.345581 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.345635 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.348350 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:33505: BlockingRecv error: recv got EOF from 127.24.11.129:33505 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:40.501977 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:40.496448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:42725)
0928 06:31:40.496636 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:40.496651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:40.496732 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:40.497000 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:40.497001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:40.497192 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:40.497239 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.497241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.498019 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.498021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.498565 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.498567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.498954 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.498955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.499014 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.499602 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:40.499606 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:40.501886 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
W20260928 06:31:40.502068 1336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:40.496598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:42725 (local address 127.24.11.190:41567)
0928 06:31:40.496679 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:40.496681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:40.496709 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:31:40.496900 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:40.496902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:40.496917 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:40.496992 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:40.497352 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.497930 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.498661 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.498865 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.498953 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.502004 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:42725: BlockingRecv error: recv got EOF from 127.24.11.130:42725 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:40 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:40.645328 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:40.640354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:33939)
0928 06:31:40.640529 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:40.640539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:40.640604 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:40.640762 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:40.640763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:40.640968 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:40.641021 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.641023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.641778 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.641779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.642302 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:40.642304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.642670 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.642671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:40.642712 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.643231 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:40.643236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:40.645265 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
W20260928 06:31:40.645471 1336 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:40.640432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:33939 (local address 127.24.11.190:41567)
0928 06:31:40.640522 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:40.640524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:40.640588 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:31:40.640675 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:40.640676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:40.640690 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:40.640736 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:40.641124 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.641658 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.642407 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:40.642593 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:40.642657 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:40.645380 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:33939: BlockingRecv error: recv got EOF from 127.24.11.131:33939 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:41.354049 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:41.348851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:35699)
0928 06:31:41.348996 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:41.349006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:41.349056 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:41.349260 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:41.349261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:41.349435 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:41.349483 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.349485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.350261 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.350262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.350792 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.350794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.351196 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.351197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.351241 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.351739 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:41.351744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:41.353987 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260928 06:31:41.354235 1340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:41.348994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:35699 (local address 127.24.11.190:41567)
0928 06:31:41.349124 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:41.349126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:41.349136 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:41.349162 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:41.349162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:41.349172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:41.349234 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:41.349642 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.350172 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.350906 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.351123 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.351182 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.354142 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:35699: BlockingRecv error: recv got EOF from 127.24.11.129:35699 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:41.507490 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:41.502534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:33013)
0928 06:31:41.502727 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:41.502780 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:41.502846 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:41.503000 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:41.503001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:41.503163 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:41.503210 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.503212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.504091 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.504092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.504643 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.504644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.505057 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.505057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.505101 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.505703 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:41.505708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:41.507433 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
W20260928 06:31:41.507668 1340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:41.502625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33013 (local address 127.24.11.190:41567)
0928 06:31:41.502714 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:41.502716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:41.502881 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:31:41.502912 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:41.502912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:41.502924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:41.502982 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:41.503306 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.503988 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.504735 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.504982 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.505043 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.507578 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:33013: BlockingRecv error: recv got EOF from 127.24.11.130:33013 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:41 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:41.650712 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:41.645869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:60955)
0928 06:31:41.646073 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:41.646084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:41.646157 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:41.646345 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:41.646346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:41.646538 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:41.646585 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.646587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.647254 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.647255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.647798 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:41.647800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.648179 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.648180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:41.648239 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.648770 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:41.648776 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:41.650614 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
W20260928 06:31:41.650760 1340 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:41.645981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:60955 (local address 127.24.11.190:41567)
0928 06:31:41.646057 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:41.646059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:41.646137 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:31:41.646224 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:41.646225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:41.646261 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:41.646338 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:41.646670 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.647167 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.647890 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:41.648090 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:41.648176 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:41.650695 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:60955: BlockingRecv error: recv got EOF from 127.24.11.131:60955 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:42.359714 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:42.354655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:58985)
0928 06:31:42.354895 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:42.354909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:42.354984 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:42.355128 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:42.355129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:42.355302 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:42.355353 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.355355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.356020 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.356021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.356680 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.356683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.357132 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.357133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.357193 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.357667 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:42.357671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:42.359613 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
W20260928 06:31:42.359769 1344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:42.354789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58985 (local address 127.24.11.190:41567)
0928 06:31:42.354951 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:42.354953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:42.354963 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:42.355038 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:42.355038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:42.355048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:42.355110 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:42.355418 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.355946 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.356776 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.357044 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.357123 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.359695 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:58985: BlockingRecv error: recv got EOF from 127.24.11.129:58985 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:42.512815 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:42.507966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:51511)
0928 06:31:42.508171 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:42.508181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:42.508239 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:42.508383 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:42.508384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:42.508561 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:42.508605 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.508606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.509547 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.509548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.510096 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.510098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.510417 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.510418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.510460 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.510899 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:42.510903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:42.512718 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":126,"threads_started":1}
W20260928 06:31:42.512868 1344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:42.508073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:51511 (local address 127.24.11.190:41567)
0928 06:31:42.508133 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:42.508135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:42.508238 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:31:42.508291 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:42.508292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:42.508303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:42.508362 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:42.508694 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.509436 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.510173 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.510352 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.510406 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.512806 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:51511: BlockingRecv error: recv got EOF from 127.24.11.130:51511 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:42 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:42.656561 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:42.651385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:43133)
0928 06:31:42.651537 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:42.651550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:42.651620 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:42.651816 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:42.651816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:42.651997 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:42.652045 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.652047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.652673 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.652673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.653191 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:42.653193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.653610 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.653612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:42.653673 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.654359 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:42.654364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:42.656452 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
W20260928 06:31:42.656666 1344 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:42.651504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:43133 (local address 127.24.11.190:41567)
0928 06:31:42.651549 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:42.651551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:42.651621 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:31:42.651718 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:42.651719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:42.651735 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:42.651790 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:42.652115 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.652606 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.653289 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:42.653483 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:42.653565 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:42.656592 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:43133: BlockingRecv error: recv got EOF from 127.24.11.131:43133 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:43.365361 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:43.360256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:32803)
0928 06:31:43.360439 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:43.360452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:43.360516 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:43.360664 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:43.360665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:43.360839 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:43.360888 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.360890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.361793 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.361794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.362323 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.362325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.362775 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.362776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.362836 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.363305 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:43.363309 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:43.365287 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":118,"threads_started":1}
W20260928 06:31:43.365579 1348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:43.360376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:32803 (local address 127.24.11.190:41567)
0928 06:31:43.360530 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:43.360532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:43.360540 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:43.360574 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:43.360575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:43.360584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:43.360657 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:43.360986 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.361651 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.362401 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.362655 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.362710 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.365464 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:32803: BlockingRecv error: recv got EOF from 127.24.11.129:32803 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:43.518193 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:43.513312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:53611)
0928 06:31:43.513493 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:43.513508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:43.513600 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:43.513789 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:43.513790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:43.513989 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:43.514033 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.514035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.514710 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.514711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.515254 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.515256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.515665 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.515666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.515734 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.516217 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:43.516222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:43.518132 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260928 06:31:43.518301 1348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:43.513439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:53611 (local address 127.24.11.190:41567)
0928 06:31:43.513553 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:43.513556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:43.513577 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:31:43.513694 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:43.513695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:43.513707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:43.513782 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:43.514120 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.514622 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.515366 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.515566 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.515648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.518232 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:53611: BlockingRecv error: recv got EOF from 127.24.11.130:53611 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:43 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:43.669373 1350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:43.657280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:40411)
0928 06:31:43.657439 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:43.657449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:43.657531 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:43.657714 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:43.657715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:43.657932 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:43.657979 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.657981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.658669 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.658670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.659277 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:43.659278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.659711 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.659712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:43.659779 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.667344 (+ 7565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:43.667350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:43.669277 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260928 06:31:43.669433 1348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:43.657431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40411 (local address 127.24.11.190:41567)
0928 06:31:43.657508 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:43.657510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:43.657521 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:43.657620 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:43.657621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:43.657642 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:43.657708 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:43.658080 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.658579 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.659375 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:43.659565 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:43.659709 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:43.669372 (+ 9663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:40411: BlockingRecv error: recv got EOF from 127.24.11.131:40411 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:44.370705 1351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:44.365900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:60621)
0928 06:31:44.366056 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:44.366067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:44.366119 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:44.366257 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:44.366258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:44.366437 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:44.366482 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.366483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.367211 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.367212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.367781 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.367784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.368138 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.368139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.368188 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.368746 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:44.368752 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:44.370633 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
W20260928 06:31:44.370927 1352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:44.365977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60621 (local address 127.24.11.190:41567)
0928 06:31:44.366120 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:44.366122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:44.366150 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:31:44.366171 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:44.366172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:44.366182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:44.366237 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:44.366579 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.367118 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.367869 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.368062 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.368120 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.370848 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:60621: BlockingRecv error: recv got EOF from 127.24.11.129:60621 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:44.523510 1353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:44.518666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:59697)
0928 06:31:44.518838 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:44.518849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:44.518934 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:44.519093 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:44.519094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:44.519281 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:44.519327 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.519329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.520001 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.520003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.520540 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.520542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.520918 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.520919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.520976 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.521492 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:44.521497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:44.523374 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
W20260928 06:31:44.523538 1352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:44.518749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:59697 (local address 127.24.11.190:41567)
0928 06:31:44.518822 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:44.518824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:44.518946 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:31:44.518997 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:44.518998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:44.519012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:44.519088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:44.519410 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.519917 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.520631 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.520831 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.520925 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.523470 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:59697: BlockingRecv error: recv got EOF from 127.24.11.130:59697 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:44 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:44.674840 1354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:44.669921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:41435)
0928 06:31:44.670104 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:44.670114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:44.670170 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:44.670309 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:44.670310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:44.670478 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:44.670524 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.670525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.671318 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.671320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.671988 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:44.671991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.672314 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.672316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:44.672360 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.672849 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:44.672855 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:44.674713 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
W20260928 06:31:44.674870 1352 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:44.670009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:41435 (local address 127.24.11.190:41567)
0928 06:31:44.670071 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:44.670073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:44.670159 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:31:44.670217 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:44.670218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:44.670232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:44.670290 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:44.670588 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.671198 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.672058 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:44.672239 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:44.672295 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:44.674804 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:41435: BlockingRecv error: recv got EOF from 127.24.11.131:41435 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:45.376456 1355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:45.371205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:53021)
0928 06:31:45.371391 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:45.371402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:45.371467 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:45.371598 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:45.371599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:45.371773 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:45.371821 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.371822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.372590 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.372592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.373125 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.373127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.373592 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.373593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.373641 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.374235 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:45.374241 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:45.376374 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
W20260928 06:31:45.376614 1356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:45.371329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:53021 (local address 127.24.11.190:41567)
0928 06:31:45.371438 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:45.371440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:45.371454 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:31:45.371520 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:45.371520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:45.371530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:45.371587 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:45.371890 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.372488 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.373234 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.373457 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.373524 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.376537 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:53021: BlockingRecv error: recv got EOF from 127.24.11.129:53021 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":52,"threads_started":1}
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:45.528872 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:45.524009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:33427)
0928 06:31:45.524138 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:45.524152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:45.524215 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:45.524379 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:45.524380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:45.524565 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:45.524609 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.524611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.525339 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.525340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.525927 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.525930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.526309 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.526310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.526375 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.526881 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:45.526888 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:45.528774 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
W20260928 06:31:45.528925 1356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:45.524123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33427 (local address 127.24.11.190:41567)
0928 06:31:45.524199 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:45.524201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:45.524212 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:45.524291 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:45.524292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:45.524305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:45.524375 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:45.524699 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.525238 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.526022 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.526223 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.526299 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.528861 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:33427: BlockingRecv error: recv got EOF from 127.24.11.130:33427 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:45 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:45.680476 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:45.675424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:52061)
0928 06:31:45.675677 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:45.675686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:45.675747 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:45.675969 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:45.675970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:45.676226 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:45.676281 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.676284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.676982 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.676983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.677584 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:45.677587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.677953 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.677954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:45.678011 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.678551 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:45.678556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:45.680377 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
W20260928 06:31:45.680519 1356 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:45.675528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52061 (local address 127.24.11.190:41567)
0928 06:31:45.675612 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:45.675614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:45.675754 (+ 140us) server_negotiation.cc:407] Connection header received
0928 06:31:45.675839 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:45.675840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:45.675856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:45.675949 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:45.676369 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.676890 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.677662 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:45.677863 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:45.677951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:45.680456 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:52061: BlockingRecv error: recv got EOF from 127.24.11.131:52061 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:46.381803 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:46.377006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:49607)
0928 06:31:46.377188 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:46.377199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:46.377255 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:46.377407 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:46.377408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:46.377669 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:46.377716 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.377718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.378440 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.378440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.378967 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.378969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.379296 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.379296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.379334 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.379792 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:46.379797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:46.381706 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
W20260928 06:31:46.381860 1360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:46.377117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:49607 (local address 127.24.11.190:41567)
0928 06:31:46.377245 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:46.377247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:46.377258 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:31:46.377303 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:46.377304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:46.377315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:46.377376 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:46.377845 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.378364 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.379041 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.379230 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.379282 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.381786 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:49607: BlockingRecv error: recv got EOF from 127.24.11.129:49607 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:46.534219 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:46.529370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:33221)
0928 06:31:46.529577 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:46.529591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:46.529686 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:46.529893 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:46.529894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:46.530067 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:46.530110 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.530111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.530786 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.530787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.531322 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.531324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.531701 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.531702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.531803 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.532314 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:46.532319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:46.534120 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260928 06:31:46.534272 1360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:46.529498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33221 (local address 127.24.11.190:41567)
0928 06:31:46.529595 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:46.529597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:46.529666 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:31:46.529786 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:46.529787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:46.529800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:46.529870 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:46.530200 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.530705 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.531428 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.531627 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.531704 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.534202 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:33221: BlockingRecv error: recv got EOF from 127.24.11.130:33221 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:46 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:46.685720 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:46.680999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:34451)
0928 06:31:46.681174 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:46.681184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:46.681238 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:46.681355 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:46.681356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:46.681576 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:46.681641 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.681643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.682313 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.682314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.682865 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:46.682867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.683252 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.683253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:46.683296 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.683793 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:46.683797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:46.685650 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
W20260928 06:31:46.685813 1360 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:46.681117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:34451 (local address 127.24.11.190:41567)
0928 06:31:46.681188 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:46.681190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:46.681225 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:31:46.681283 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:46.681283 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:46.681296 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:46.681351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:46.681709 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.682219 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.682971 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:46.683176 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:46.683240 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:46.685746 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:34451: BlockingRecv error: recv got EOF from 127.24.11.131:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:47.388177 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:47.382385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:52919)
0928 06:31:47.382508 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:47.382519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:47.382581 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:47.382761 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:47.382763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:47.382980 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:47.383039 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.383042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.384114 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.384115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.384664 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.384665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.385194 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.385195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.385234 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.385786 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:47.385792 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:47.388062 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260928 06:31:47.388375 1364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:47.382432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:52919 (local address 127.24.11.190:41567)
0928 06:31:47.382543 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:47.382545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:47.382559 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:31:47.382663 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:47.382664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:47.382678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:47.382745 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:47.383138 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.384024 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.384748 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.385108 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.385199 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.388198 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:52919: BlockingRecv error: recv got EOF from 127.24.11.129:52919 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:47.539333 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:47.534749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:39655)
0928 06:31:47.534998 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:47.535025 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:47.535092 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:47.535224 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:47.535225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:47.535396 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:47.535443 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.535445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.536057 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.536058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.536585 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.536587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.536906 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.536907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.536947 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.537399 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:47.537404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:47.539276 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
W20260928 06:31:47.539469 1364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:47.534809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:39655 (local address 127.24.11.190:41567)
0928 06:31:47.535000 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:47.535002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:47.535092 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:31:47.535140 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:47.535141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:47.535151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:47.535202 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:47.535505 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.535983 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.536660 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.536837 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.536886 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.539389 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:39655: BlockingRecv error: recv got EOF from 127.24.11.130:39655 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:47 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.131@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:47.690689 1366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:47.686223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.131:58523)
0928 06:31:47.686375 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:47.686394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:47.686450 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:47.686586 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:47.686587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:47.686751 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:47.686797 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.686799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.687440 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.687441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.687971 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:47.687973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.688327 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.688328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:47.688369 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.688802 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:47.688806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:47.690630 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
W20260928 06:31:47.690881 1364 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:47.686322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:58523 (local address 127.24.11.190:41567)
0928 06:31:47.686403 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:47.686409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:47.686444 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:31:47.686507 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:47.686508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:47.686520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:47.686564 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:47.686873 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.687366 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.688054 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:47.688236 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:47.688326 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:47.690798 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.131:58523: BlockingRecv error: recv got EOF from 127.24.11.131:58523 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 06:31:48 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:48 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.129@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:48.393628 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:48.388718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.129:36415)
0928 06:31:48.388878 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:48.388901 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:48.388964 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:48.389121 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:48.389121 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:48.389285 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:48.389343 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.389345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.390121 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.390122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.390672 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.390674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.391113 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.391114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.391165 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.391659 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:48.391664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:48.393511 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
W20260928 06:31:48.393680 1368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:48.388849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:36415 (local address 127.24.11.190:41567)
0928 06:31:48.388983 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:48.388986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:48.389001 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:31:48.389025 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:48.389025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:48.389036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:48.389102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:48.389423 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.390043 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.390752 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.390982 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.391039 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.393627 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.129:36415: BlockingRecv error: recv got EOF from 127.24.11.129:36415 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
Sep 28 06:31:48 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:31:48 dist-test-slave-38pg krb5kdc[306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.11.130@KRBTEST.COM for oryx/127.24.11.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:31:48.544651 1369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:48.539846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:41567 (local address 127.24.11.130:35193)
0928 06:31:48.540035 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:48.540047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:48.540130 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:48.540295 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:48.540296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:48.540484 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:48.540530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.540531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.541268 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.541269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.541828 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.541831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.542225 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.542226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.542285 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.542803 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:48.542808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:48.544553 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.11.190:41567: Server oryx/127.24.11.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
W20260928 06:31:48.544694 1368 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:48.539947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35193 (local address 127.24.11.190:41567)
0928 06:31:48.540017 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:48.540019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:48.540111 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:31:48.540198 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:48.540199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:48.540212 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:48.540289 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:48.540613 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.541181 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.541924 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.542138 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.542224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.544633 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.11.130:35193: BlockingRecv error: recv got EOF from 127.24.11.130:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:31:48.592487 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 889
I20260928 06:31:48.597278 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1041
I20260928 06:31:48.601573 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1176
I20260928 06:31:48.605671 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 819
2026-09-28T06:31:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17527 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:48Z Disabled control of system clock
I20260928 06:31:48.633808 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:33017
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:31:48.714349 1373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:48.714581 1373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:48.714601 1373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:48.715826 1373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:48.715875 1373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:48.715888 1373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:48.715901 1373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:48.715914 1373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:48.717276 1373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:33017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:48.717566 1373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:48.717905 1373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:48.723295 1381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:48.723275 1378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:48.723356 1373 server_base.cc:1034] running on GCE node
W20260928 06:31:48.723270 1379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:48.723855 1373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:48.724174 1373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:48.725319 1373 hybrid_clock.cc:648] HybridClock initialized: now 1790577108725306 us; error 28 us; skew 500 ppm
I20260928 06:31:48.732790 1373 webserver.cc:460] Webserver started at http://127.24.11.190:40621/ using document root <none> and password file <none>
I20260928 06:31:48.733042 1373 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:48.733090 1373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:48.733196 1373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:48.734423 1373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "d6712f10e5a0e29011531a8393ae7b68"
server_key_iv: "6115b022969f9d006edd13ad3ac20552"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.734757 1373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "d6712f10e5a0e29011531a8393ae7b68"
server_key_iv: "6115b022969f9d006edd13ad3ac20552"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.736123 1373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:31:48.736912 1387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:48.737196 1373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:48.737285 1373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "d6712f10e5a0e29011531a8393ae7b68"
server_key_iv: "6115b022969f9d006edd13ad3ac20552"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.737337 1373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:48.767673 1373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:48.767983 1373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:48.768092 1373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:48.772081 1373 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:33017
I20260928 06:31:48.772152 1440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:33017 every 8 connection(s)
I20260928 06:31:48.772521 1373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:48.773290 1441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:48.776122 1441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: Bootstrap starting.
I20260928 06:31:48.776758 1441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:48.777062 1441 log.cc:824] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:48.777982 1441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: No bootstrap required, opened a new log
I20260928 06:31:48.778714 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1373
I20260928 06:31:48.778793 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:48.779053 24622 external_mini_cluster.cc:1428] Setting key fc5b053acf8ac8ba3b7930a9b9845142
I20260928 06:31:48.780102 1441 raft_consensus.cc:348] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } }
I20260928 06:31:48.780287 1441 raft_consensus.cc:374] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:48.780354 1441 raft_consensus.cc:729] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7b8dd1f75c440d8fb3b5cd9ef1824f, State: Initialized, Role: FOLLOWER
I20260928 06:31:48.780519 1441 consensus_queue.cc:260] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } }
I20260928 06:31:48.780611 1441 raft_consensus.cc:388] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:48.780644 1441 raft_consensus.cc:482] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:48.780687 1441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:48.781503 1441 raft_consensus.cc:504] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } }
I20260928 06:31:48.781633 1441 leader_election.cc:302] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be7b8dd1f75c440d8fb3b5cd9ef1824f; no voters:
I20260928 06:31:48.781816 1441 leader_election.cc:288] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:48.781924 1446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:48.782109 1446 raft_consensus.cc:686] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [term 1 LEADER]: Becoming Leader. State: Replica: be7b8dd1f75c440d8fb3b5cd9ef1824f, State: Running, Role: LEADER
I20260928 06:31:48.782150 1441 sys_catalog.cc:564] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:48.783566 1446 consensus_queue.cc:237] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } }
I20260928 06:31:48.784256 1448 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } } }
I20260928 06:31:48.784240 1447 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [sys.catalog]: SysCatalogTable state changed. Reason: New leader be7b8dd1f75c440d8fb3b5cd9ef1824f. Latest consensus state: current_term: 1 leader_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7b8dd1f75c440d8fb3b5cd9ef1824f" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 33017 } } }
I20260928 06:31:48.785172 1447 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:48.785168 1448 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f [sys.catalog]: This master's current role is: LEADER
W20260928 06:31:48.784802 1460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:31:48.785382 1460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:31:48.785522 1463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:48.785775 1463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:48.786172 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:48.779746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.24.11.190:33017)
0928 06:31:48.779997 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:48.780005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:48.780031 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:31:48.780130 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:48.780136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:48.780190 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:48.780338 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:48.780569 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.782379 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.783488 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.784080 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.784212 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.785317 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:48.785320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:48.785320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:48.785560 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:48.785607 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:48.785624 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:48.785689 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:48.785820 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:31:48.785927 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260928 06:31:48.787657 1463 catalog_manager.cc:1347] Generated new cluster ID: d0d2a8f8ca4541c1841828a2f6add993
I20260928 06:31:48.787715 1463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:48.806598 1463 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:48.807343 1463 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:48.823010 1463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be7b8dd1f75c440d8fb3b5cd9ef1824f: Generated new TSK 0
I20260928 06:31:48.823251 1463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:48.831969 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--builtin_ntp_servers=127.24.11.148:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:31:48.915935 1465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:48.916162 1465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:48.916183 1465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:48.917454 1465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:48.917498 1465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:48.917570 1465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:48.918957 1465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:48.919214 1465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:48.919530 1465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:48.924682 1470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:48.924842 1465 server_base.cc:1034] running on GCE node
W20260928 06:31:48.924729 1473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:48.924696 1471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:48.925194 1465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:48.925421 1465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:48.926622 1465 hybrid_clock.cc:648] HybridClock initialized: now 1790577108926610 us; error 39 us; skew 500 ppm
I20260928 06:31:48.927963 1465 webserver.cc:460] Webserver started at http://127.24.11.129:43741/ using document root <none> and password file <none>
I20260928 06:31:48.928181 1465 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:48.928227 1465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:48.928328 1465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:48.929471 1465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "e0cbce8156a5416abda87586ff16e0ff"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "929a3eb7aa540add6990633f061a9a45"
server_key_iv: "9a35b1741bf46b183d8e32f66f33b075"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.929883 1465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "e0cbce8156a5416abda87586ff16e0ff"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "929a3eb7aa540add6990633f061a9a45"
server_key_iv: "9a35b1741bf46b183d8e32f66f33b075"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.931222 1465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:48.931936 1479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:48.932089 1465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:48.932173 1465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "e0cbce8156a5416abda87586ff16e0ff"
format_stamp: "Formatted at 2026-09-28 06:31:48 on dist-test-slave-38pg"
server_key: "929a3eb7aa540add6990633f061a9a45"
server_key_iv: "9a35b1741bf46b183d8e32f66f33b075"
server_key_version: "encryptionkey@0"
I20260928 06:31:48.932230 1465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:48.967868 1465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:48.968179 1465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:48.968299 1465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:48.968525 1465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:48.968835 1465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:48.968870 1465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:48.968894 1465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:48.968906 1465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:48.974570 1465 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:37149
I20260928 06:31:48.974648 1592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:37149 every 8 connection(s)
I20260928 06:31:48.975004 1465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:48.977262 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1465
I20260928 06:31:48.977370 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:48.977591 24622 external_mini_cluster.cc:1428] Setting key b8b0149d807e20f743ba49152c30b06f
I20260928 06:31:48.978451 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--builtin_ntp_servers=127.24.11.148:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:31:48.980618 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:48.975796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:39461 (local address 127.24.11.190:33017)
0928 06:31:48.975897 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:48.975899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:48.976112 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:31:48.976669 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:48.976670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:48.976687 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:48.976738 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:48.977415 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.978022 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.979095 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.979320 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.979406 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.980349 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:48.980350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:48.980350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:48.980355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:48.980370 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:48.980374 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:48.980430 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:48.980533 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:31:48.980557 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 06:31:48.980962 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:48.975779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:33017 (local address 127.24.11.129:39461)
0928 06:31:48.975992 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:48.976026 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:48.976552 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:48.976815 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:48.976818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:48.976827 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:48.977309 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.977313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.978130 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.978131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.979003 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:48.979005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:48.979413 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:48.979414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:48.979584 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:48.980183 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:48.980183 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:48.980224 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:48.980255 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:48.980444 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:48.980450 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:48.980455 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:48.980538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:31:48.980655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260928 06:31:48.981570 1593 heartbeater.cc:347] Connected to a master server at 127.24.11.190:33017
I20260928 06:31:48.981684 1593 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:48.981871 1593 heartbeater.cc:511] Master 127.24.11.190:33017 requested a full tablet report, sending...
I20260928 06:31:48.982414 1404 ts_manager.cc:194] Registered new tserver with Master: e0cbce8156a5416abda87586ff16e0ff (127.24.11.129:37149)
I20260928 06:31:48.983204 1404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.129:39461
W20260928 06:31:49.061651 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.061885 1596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.061919 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.063076 1596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.063141 1596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.063182 1596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:49.064502 1596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.064760 1596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.065097 1596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.070015 1601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.070051 1602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.070075 1604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.070274 1596 server_base.cc:1034] running on GCE node
I20260928 06:31:49.070485 1596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.070678 1596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.071815 1596 hybrid_clock.cc:648] HybridClock initialized: now 1790577109071803 us; error 31 us; skew 500 ppm
I20260928 06:31:49.073156 1596 webserver.cc:460] Webserver started at http://127.24.11.130:35721/ using document root <none> and password file <none>
I20260928 06:31:49.073400 1596 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.073457 1596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.073616 1596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.074718 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "20910ad6ea6f42429fad726735d1f2a1"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "35eb885b7040e4f9bdec145f68585b1c"
server_key_iv: "d4c0345cdcafd8fab7893676d4864873"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.075100 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "20910ad6ea6f42429fad726735d1f2a1"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "35eb885b7040e4f9bdec145f68585b1c"
server_key_iv: "d4c0345cdcafd8fab7893676d4864873"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.076426 1596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:49.077198 1610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.077392 1596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:49.077507 1596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "20910ad6ea6f42429fad726735d1f2a1"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "35eb885b7040e4f9bdec145f68585b1c"
server_key_iv: "d4c0345cdcafd8fab7893676d4864873"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.077626 1596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.091020 1596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.091346 1596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.091507 1596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.091749 1596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:49.092092 1596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:49.092144 1596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.092193 1596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:49.092226 1596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.097970 1596 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:43773
I20260928 06:31:49.098057 1724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:43773 every 8 connection(s)
I20260928 06:31:49.098376 1596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:49.103583 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.099166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:46001 (local address 127.24.11.190:33017)
0928 06:31:49.099222 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.099224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.099466 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:31:49.099972 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.099973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.099983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.100049 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.100725 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.101241 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.102224 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.102407 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.102466 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:49.103337 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.103338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.103338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.103342 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.103357 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.103361 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.103412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.103497 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.103522 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:31:49.103758 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.099137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:33017 (local address 127.24.11.130:46001)
0928 06:31:49.099340 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:49.099389 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:49.099854 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:49.100114 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:49.100117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:49.100125 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:49.100631 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:49.100635 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.101340 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.101341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:49.102148 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:49.102150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.102489 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.102490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:49.102598 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:49.103194 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:49.103194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:49.103225 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:49.103248 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:49.103420 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:49.103423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:49.103425 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:49.103477 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:31:49.103558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260928 06:31:49.103900 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1596
I20260928 06:31:49.104029 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:49.104233 24622 external_mini_cluster.cc:1428] Setting key 1fc1a2715a6aced397c63e7542727136
I20260928 06:31:49.104215 1725 heartbeater.cc:347] Connected to a master server at 127.24.11.190:33017
I20260928 06:31:49.104321 1725 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:49.104504 1725 heartbeater.cc:511] Master 127.24.11.190:33017 requested a full tablet report, sending...
I20260928 06:31:49.104890 1404 ts_manager.cc:194] Registered new tserver with Master: 20910ad6ea6f42429fad726735d1f2a1 (127.24.11.130:43773)
I20260928 06:31:49.105212 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--builtin_ntp_servers=127.24.11.148:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 06:31:49.105273 1404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.130:46001
W20260928 06:31:49.186786 1728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.187036 1728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.187060 1728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.188253 1728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.188297 1728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.188324 1728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:49.189721 1728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:33017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.189945 1728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.190227 1728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.195391 1734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.195391 1736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.195422 1733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.195691 1728 server_base.cc:1034] running on GCE node
I20260928 06:31:49.195966 1728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.196257 1728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.197412 1728 hybrid_clock.cc:648] HybridClock initialized: now 1790577109197402 us; error 34 us; skew 500 ppm
I20260928 06:31:49.198863 1728 webserver.cc:460] Webserver started at http://127.24.11.131:45463/ using document root <none> and password file <none>
I20260928 06:31:49.199189 1728 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.199246 1728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.199482 1728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.204104 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "459ed5fc86c148078efb20962a4f9efb"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "5c0fc127b058d2c5c35371f27cf21ad4"
server_key_iv: "5f48d9507a1e41de3615c170c77b1cc2"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.204571 1728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "459ed5fc86c148078efb20962a4f9efb"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "5c0fc127b058d2c5c35371f27cf21ad4"
server_key_iv: "5f48d9507a1e41de3615c170c77b1cc2"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.206147 1728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 06:31:49.206979 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.207192 1728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:49.207291 1728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "459ed5fc86c148078efb20962a4f9efb"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "5c0fc127b058d2c5c35371f27cf21ad4"
server_key_iv: "5f48d9507a1e41de3615c170c77b1cc2"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.207376 1728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.227119 1728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.227504 1728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.227662 1728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.227941 1728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:49.228322 1728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:49.228381 1728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.228418 1728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:49.228444 1728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.234620 1728 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:43595
I20260928 06:31:49.234663 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:43595 every 8 connection(s)
I20260928 06:31:49.235059 1728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:49.240343 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1728
I20260928 06:31:49.240474 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:49.240640 24622 external_mini_cluster.cc:1428] Setting key 7625eb0d9a72f8efe9795bd856d830fe
I20260928 06:31:49.240914 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.235964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42357 (local address 127.24.11.190:33017)
0928 06:31:49.236007 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.236009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.236276 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:31:49.236801 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.236802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.236813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.236886 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.237689 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.238210 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.239223 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.239423 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.239489 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:49.240527 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.240529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.240529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.240535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.240550 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.240555 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.240626 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.240725 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.240760 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:31:49.241187 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.235871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:33017 (local address 127.24.11.131:42357)
0928 06:31:49.236111 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:49.236155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:49.236653 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:49.236992 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:49.236996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:49.237005 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:49.237594 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:49.237604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.238296 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.238297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:49.239127 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:49.239128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:49.239507 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:49.239508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:49.239624 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:49.240345 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:49.240346 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:49.240378 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:49.240406 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:49.240627 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:49.240634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:49.240637 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:49.240729 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:31:49.240847 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260928 06:31:49.241717 1856 heartbeater.cc:347] Connected to a master server at 127.24.11.190:33017
I20260928 06:31:49.241820 1856 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:49.242029 1856 heartbeater.cc:511] Master 127.24.11.190:33017 requested a full tablet report, sending...
I20260928 06:31:49.242393 1404 ts_manager.cc:194] Registered new tserver with Master: 459ed5fc86c148078efb20962a4f9efb (127.24.11.131:43595)
I20260928 06:31:49.242769 1404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.131:42357
I20260928 06:31:49.252921 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:31:49.255347 1445 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:49.254705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.24.11.190:33017)
0928 06:31:49.254801 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.254804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.254919 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:31:49.255027 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.255028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.255042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.255089 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.255263 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45070: BlockingRecv error: recv got EOF from 127.0.0.1:45070 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 06:31:49.255484 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1465
I20260928 06:31:49.260461 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1596
I20260928 06:31:49.264932 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1728
I20260928 06:31:49.269428 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1373
2026-09-28T06:31:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (663 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:49Z Disabled control of system clock
I20260928 06:31:49.297516 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43273
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:31:49.381029 1867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.381278 1867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.381302 1867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.382550 1867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:49.382606 1867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.382619 1867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.382633 1867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:49.382647 1867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:49.384109 1867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.384361 1867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.384626 1867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.389716 1873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.389707 1875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.389925 1867 server_base.cc:1034] running on GCE node
W20260928 06:31:49.389707 1872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.390332 1867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.390614 1867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.391806 1867 hybrid_clock.cc:648] HybridClock initialized: now 1790577109391792 us; error 45 us; skew 500 ppm
I20260928 06:31:49.393167 1867 webserver.cc:460] Webserver started at http://127.24.11.190:46209/ using document root <none> and password file <none>
I20260928 06:31:49.393440 1867 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.393519 1867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.393720 1867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.394912 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "9bbeeb2fcd7a497c87400ce372292825"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "0daebf865d758ff2905b8830b6407430"
server_key_iv: "281a0a679bc6094d42d3060c06170b08"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.395327 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "9bbeeb2fcd7a497c87400ce372292825"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "0daebf865d758ff2905b8830b6407430"
server_key_iv: "281a0a679bc6094d42d3060c06170b08"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.396857 1867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:49.397773 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.397971 1867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:49.398084 1867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "9bbeeb2fcd7a497c87400ce372292825"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "0daebf865d758ff2905b8830b6407430"
server_key_iv: "281a0a679bc6094d42d3060c06170b08"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.398169 1867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.401897 1867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.402253 1867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.402427 1867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.406693 1867 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43273
I20260928 06:31:49.406729 1933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43273 every 8 connection(s)
I20260928 06:31:49.407100 1867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:49.407809 1934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:49.410976 1934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825: Bootstrap starting.
I20260928 06:31:49.411649 1934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:49.411993 1934 log.cc:824] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:49.412671 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1867
I20260928 06:31:49.412786 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:49.412895 1934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825: No bootstrap required, opened a new log
I20260928 06:31:49.412997 24622 external_mini_cluster.cc:1428] Setting key 278495ac775fa5d8ba71a21a9c6a5e1a
I20260928 06:31:49.415510 1934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } }
I20260928 06:31:49.415690 1934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:49.415748 1934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bbeeb2fcd7a497c87400ce372292825, State: Initialized, Role: FOLLOWER
I20260928 06:31:49.415874 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.413618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.24.11.190:43273)
0928 06:31:49.413866 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.413881 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.413907 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:31:49.414386 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.414393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.414433 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.414552 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.415305 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.415308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.415308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.415312 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.415354 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.415365 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.415422 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.415514 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.415592 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":161,"threads_started":1}
I20260928 06:31:49.415880 1934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } }
I20260928 06:31:49.416064 1934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:49.416105 1934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:49.416153 1934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:49.417187 1934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } }
I20260928 06:31:49.417290 1934 leader_election.cc:302] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bbeeb2fcd7a497c87400ce372292825; no voters:
I20260928 06:31:49.417418 1934 leader_election.cc:288] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:49.417507 1939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:49.417695 1939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [term 1 LEADER]: Becoming Leader. State: Replica: 9bbeeb2fcd7a497c87400ce372292825, State: Running, Role: LEADER
I20260928 06:31:49.417725 1934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:49.417793 1939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } }
I20260928 06:31:49.418130 1939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bbeeb2fcd7a497c87400ce372292825. Latest consensus state: current_term: 1 leader_uuid: "9bbeeb2fcd7a497c87400ce372292825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } } }
I20260928 06:31:49.418141 1940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bbeeb2fcd7a497c87400ce372292825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bbeeb2fcd7a497c87400ce372292825" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43273 } } }
I20260928 06:31:49.418259 1939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:49.418334 1940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825 [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:49.418623 1948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:49.419173 1948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:49.420738 1948 catalog_manager.cc:1347] Generated new cluster ID: f779d491304649b0990f5749c7214199
I20260928 06:31:49.420791 1948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:49.427053 1948 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:49.427282 1948 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:49.431432 1948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bbeeb2fcd7a497c87400ce372292825: Generated new TSK 0
I20260928 06:31:49.431675 1948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:31:49.433665 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--builtin_ntp_servers=127.24.11.148:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:31:49.525318 1958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.525626 1958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.525700 1958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.527004 1958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.527079 1958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.527130 1958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:49.528592 1958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.528878 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.529202 1958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.534499 1964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.534657 1958 server_base.cc:1034] running on GCE node
W20260928 06:31:49.534503 1963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.534499 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.535089 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.535434 1958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.536649 1958 hybrid_clock.cc:648] HybridClock initialized: now 1790577109536622 us; error 41 us; skew 500 ppm
I20260928 06:31:49.538394 1958 webserver.cc:460] Webserver started at http://127.24.11.129:41545/ using document root <none> and password file <none>
I20260928 06:31:49.538700 1958 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.538774 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.538938 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.540563 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "55e1c80c704a41d69e280a6ad4fc0c6c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "2861b7d6e6db21f81b878ec4e87b93f5"
server_key_iv: "1f97b09d9bd54ca5f6520193a17ce564"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.541061 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "55e1c80c704a41d69e280a6ad4fc0c6c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "2861b7d6e6db21f81b878ec4e87b93f5"
server_key_iv: "1f97b09d9bd54ca5f6520193a17ce564"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.543162 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:31:49.544230 1972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.544456 1958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:49.544566 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "55e1c80c704a41d69e280a6ad4fc0c6c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "2861b7d6e6db21f81b878ec4e87b93f5"
server_key_iv: "1f97b09d9bd54ca5f6520193a17ce564"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.544660 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.550276 1958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.550601 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.550800 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.551036 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:49.551373 1958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:49.551426 1958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.551461 1958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:49.551489 1958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.557052 1958 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:42477
I20260928 06:31:49.557157 2085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:42477 every 8 connection(s)
I20260928 06:31:49.557572 1958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:49.560956 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 1958
I20260928 06:31:49.561105 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:49.561295 24622 external_mini_cluster.cc:1428] Setting key 024b9dfcccf10bd231ada4eec251b9df
I20260928 06:31:49.561491 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.558416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:50395 (local address 127.24.11.190:43273)
0928 06:31:49.558493 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.558496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.558816 (+ 320us) server_negotiation.cc:407] Connection header received
0928 06:31:49.559569 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.559570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.559577 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.559626 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.561165 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.561167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.561168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.561174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.561191 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.561197 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.561232 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.561388 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.561406 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:31:49.562085 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.558398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43273 (local address 127.24.11.129:50395)
0928 06:31:49.558645 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:49.558716 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:49.559440 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:49.559706 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:49.559711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:49.559723 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:49.560938 (+ 1215us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:49.560939 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:49.560985 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:49.561033 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:49.561278 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:49.561283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:49.561287 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:49.561756 (+ 469us) client_negotiation.cc:240] Negotiation successful
0928 06:31:49.561841 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260928 06:31:49.562413 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--builtin_ntp_servers=127.24.11.148:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 06:31:49.562695 2086 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43273
I20260928 06:31:49.562827 2086 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:49.563022 2086 heartbeater.cc:511] Master 127.24.11.190:43273 requested a full tablet report, sending...
I20260928 06:31:49.563598 1898 ts_manager.cc:194] Registered new tserver with Master: 55e1c80c704a41d69e280a6ad4fc0c6c (127.24.11.129:42477)
W20260928 06:31:49.654088 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.654333 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.654367 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.655733 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.655793 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.655824 2089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:49.657370 2089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.657708 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.658048 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.663528 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.663522 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.663600 2089 server_base.cc:1034] running on GCE node
W20260928 06:31:49.663525 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.664054 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.664373 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.665585 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790577109665571 us; error 31 us; skew 500 ppm
I20260928 06:31:49.666996 2089 webserver.cc:460] Webserver started at http://127.24.11.130:33087/ using document root <none> and password file <none>
I20260928 06:31:49.667281 2089 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.667382 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.667543 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.668748 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "e986a16a7049450abd10df431152321c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "4d0a5720433a11399907cd075b8aac66"
server_key_iv: "268607712063ee7d7b8047fa82635898"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.669133 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "e986a16a7049450abd10df431152321c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "4d0a5720433a11399907cd075b8aac66"
server_key_iv: "268607712063ee7d7b8047fa82635898"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.670661 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:31:49.671511 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.671694 2089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:49.671806 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "e986a16a7049450abd10df431152321c"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "4d0a5720433a11399907cd075b8aac66"
server_key_iv: "268607712063ee7d7b8047fa82635898"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.671891 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.675596 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.675891 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.676038 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.676250 2089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:49.676582 2089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:49.676635 2089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.676677 2089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:49.676704 2089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.682871 2089 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37531
I20260928 06:31:49.682929 2216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37531 every 8 connection(s)
I20260928 06:31:49.683276 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:49.686650 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.684113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:40177 (local address 127.24.11.190:43273)
0928 06:31:49.684177 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.684179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.684442 (+ 263us) server_negotiation.cc:407] Connection header received
0928 06:31:49.684989 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.684990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.684996 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.685059 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.686367 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.686369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.686369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.686374 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.686392 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.686397 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.686461 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.686558 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.686575 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:49.686892 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.684112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43273 (local address 127.24.11.130:40177)
0928 06:31:49.684302 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:49.684338 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:49.684865 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:49.685143 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:49.685146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:49.685155 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:49.686194 (+ 1039us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:49.686195 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:49.686228 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:49.686262 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:49.686463 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:49.686469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:49.686473 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:49.686570 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:31:49.686637 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260928 06:31:49.687445 2217 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43273
I20260928 06:31:49.687551 2217 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:49.687712 2217 heartbeater.cc:511] Master 127.24.11.190:43273 requested a full tablet report, sending...
I20260928 06:31:49.688160 1898 ts_manager.cc:194] Registered new tserver with Master: e986a16a7049450abd10df431152321c (127.24.11.130:37531)
I20260928 06:31:49.688576 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2089
I20260928 06:31:49.688707 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:49.688911 24622 external_mini_cluster.cc:1428] Setting key 67207d0a69103b13b32de72d71a0864c
I20260928 06:31:49.689841 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--builtin_ntp_servers=127.24.11.148:34911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:31:49.777629 2220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:49.777910 2220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:49.777951 2220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:49.779290 2220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:49.779376 2220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:49.779424 2220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:49.780974 2220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:34911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:49.781297 2220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:49.781682 2220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:49.787345 2225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.787402 2226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:49.787402 2228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:49.787536 2220 server_base.cc:1034] running on GCE node
I20260928 06:31:49.788070 2220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:49.788371 2220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:49.789642 2220 hybrid_clock.cc:648] HybridClock initialized: now 1790577109789615 us; error 45 us; skew 500 ppm
I20260928 06:31:49.791210 2220 webserver.cc:460] Webserver started at http://127.24.11.131:34693/ using document root <none> and password file <none>
I20260928 06:31:49.791495 2220 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:49.791574 2220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:49.791739 2220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:49.792917 2220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "e9652448995a4755b9afcea7616d3869"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "ced26f40cc3e10f3bce04cdd89c20504"
server_key_iv: "803c8186ce2b1404c93c98611c9d52d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.793306 2220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "e9652448995a4755b9afcea7616d3869"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "ced26f40cc3e10f3bce04cdd89c20504"
server_key_iv: "803c8186ce2b1404c93c98611c9d52d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.794941 2220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:31:49.795783 2234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.796000 2220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:49.796129 2220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "e9652448995a4755b9afcea7616d3869"
format_stamp: "Formatted at 2026-09-28 06:31:49 on dist-test-slave-38pg"
server_key: "ced26f40cc3e10f3bce04cdd89c20504"
server_key_iv: "803c8186ce2b1404c93c98611c9d52d6"
server_key_version: "encryptionkey@0"
I20260928 06:31:49.796216 2220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:49.800007 2220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:49.800328 2220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:49.800482 2220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:49.800699 2220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:49.801038 2220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:49.801093 2220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.801136 2220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:49.801164 2220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:49.807745 2220 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37345
I20260928 06:31:49.807852 2347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37345 every 8 connection(s)
I20260928 06:31:49.808538 2220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:49.811815 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.809481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40967 (local address 127.24.11.190:43273)
0928 06:31:49.809597 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.809599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.809794 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:31:49.810305 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.810306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.810312 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.810375 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.811559 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:49.811561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:49.811561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:31:49.811565 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:49.811580 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:31:49.811584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:49.811668 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:49.811734 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:31:49.811747 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 06:31:49.812036 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:49.809424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43273 (local address 127.24.11.131:40967)
0928 06:31:49.809659 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:49.809695 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:49.810176 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:49.810454 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:49.810458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:49.810466 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:49.811396 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:31:49.811397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:49.811430 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:31:49.811461 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:49.811679 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:49.811682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:49.811685 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:49.811745 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:31:49.811805 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":132,"threads_started":1}
I20260928 06:31:49.812655 2348 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43273
I20260928 06:31:49.812772 2348 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:49.812958 2348 heartbeater.cc:511] Master 127.24.11.190:43273 requested a full tablet report, sending...
I20260928 06:31:49.813321 1898 ts_manager.cc:194] Registered new tserver with Master: e9652448995a4755b9afcea7616d3869 (127.24.11.131:37345)
I20260928 06:31:49.815150 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2220
I20260928 06:31:49.815259 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:49.815487 24622 external_mini_cluster.cc:1428] Setting key e4f8456ae6143ad996ca66f7a3e82f2e
I20260928 06:31:49.816550 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:31:49.818559 1938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:31:49.818081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.24.11.190:43273)
0928 06:31:49.818121 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:49.818123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:49.818252 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:31:49.818316 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:49.818316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:49.818321 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:49.818376 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:49.818476 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53654: BlockingRecv error: recv got EOF from 127.0.0.1:53654 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:31:49.818667 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1958
I20260928 06:31:49.824795 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2089
I20260928 06:31:49.829922 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2220
I20260928 06:31:49.834313 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 1867
2026-09-28T06:31:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (564 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:49 dist-test-slave-38pg krb5kdc[2358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:49 dist-test-slave-38pg krb5kdc[2358](info): set up 2 sockets
Sep 28 06:31:49 dist-test-slave-38pg krb5kdc[2358](info): commencing operation
krb5kdc: starting...
W20260928 06:31:50.871603 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:50 dist-test-slave-38pg krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:31:50.997169 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44197
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:51.077520 2371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:51.077757 2371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:51.077780 2371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:51.078872 2371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:31:51.078912 2371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:51.078927 2371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:51.078940 2371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:31:51.078953 2371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:31:51.080325 2371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:44197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:51.080519 2371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:51.080683 2371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:51.085500 2379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:51.085493 2376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.085602 2371 server_base.cc:1034] running on GCE node
W20260928 06:31:51.085493 2377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.085960 2371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:51.086268 2371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:51.087441 2371 hybrid_clock.cc:648] HybridClock initialized: now 1790577111087431 us; error 47 us; skew 500 ppm
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:51.089941 2371 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:31:51.090404 2371 webserver.cc:460] Webserver started at http://127.24.11.190:33403/ using document root <none> and password file <none>
I20260928 06:31:51.090592 2371 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:51.090632 2371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:51.090723 2371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:51.091709 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "0daf707007254264b758567b34fb99da"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "b1ebdf22479a1d4846abf9240d904821"
server_key_iv: "c5cf923d63ba47448dd280b11cf7daaa"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.092005 2371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "0daf707007254264b758567b34fb99da"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "b1ebdf22479a1d4846abf9240d904821"
server_key_iv: "c5cf923d63ba47448dd280b11cf7daaa"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.093235 2371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:31:51.094048 2386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.094224 2371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:31:51.094329 2371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "0daf707007254264b758567b34fb99da"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "b1ebdf22479a1d4846abf9240d904821"
server_key_iv: "c5cf923d63ba47448dd280b11cf7daaa"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.094415 2371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:51.104437 2371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:51.106415 2371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:51.106576 2371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:51.110550 2371 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:44197
I20260928 06:31:51.110587 2438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:44197 every 8 connection(s)
I20260928 06:31:51.110957 2371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:31:51.111958 2439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:51.111953 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2371
I20260928 06:31:51.112036 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:31:51.112252 24622 external_mini_cluster.cc:1428] Setting key 9bc1f5086db037626c81d30e27ba620b
I20260928 06:31:51.115293 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: Bootstrap starting.
I20260928 06:31:51.116024 2439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:51.116312 2439 log.cc:824] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:51.117043 2439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: No bootstrap required, opened a new log
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:51.119608 2439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } }
I20260928 06:31:51.119767 2439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:51.119866 2439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daf707007254264b758567b34fb99da, State: Initialized, Role: FOLLOWER
I20260928 06:31:51.120019 2439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } }
I20260928 06:31:51.120128 2439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:31:51.120172 2439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:31:51.120221 2439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:51.120718 2439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } }
I20260928 06:31:51.120844 2439 leader_election.cc:302] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0daf707007254264b758567b34fb99da; no voters:
I20260928 06:31:51.121018 2439 leader_election.cc:288] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:31:51.121078 2444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:51.121248 2444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [term 1 LEADER]: Becoming Leader. State: Replica: 0daf707007254264b758567b34fb99da, State: Running, Role: LEADER
I20260928 06:31:51.121277 2439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:31:51.121366 2444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } }
I20260928 06:31:51.121773 2444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0daf707007254264b758567b34fb99da. Latest consensus state: current_term: 1 leader_uuid: "0daf707007254264b758567b34fb99da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } } }
I20260928 06:31:51.121781 2445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0daf707007254264b758567b34fb99da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daf707007254264b758567b34fb99da" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44197 } } }
I20260928 06:31:51.121966 2445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:51.122169 2444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da [sys.catalog]: This master's current role is: LEADER
I20260928 06:31:51.122519 2453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:31:51.123128 2453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:31:51.124233 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.112797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.24.11.190:44197)
0928 06:31:51.112964 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.112971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.112996 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:31:51.113480 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.113487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.113621 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.113736 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:51.114528 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.115473 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.116525 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.117049 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.117182 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.119742 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:51.119746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:51.119746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:51.119758 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:51.122158 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.122641 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.122644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.122644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.122762 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.122977 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.122978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.122978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.123325 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:51.123834 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.123905 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.124039 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260928 06:31:51.124573 2453 catalog_manager.cc:1347] Generated new cluster ID: 2e9fd4ed37134fa29e24d82ecf9a417e
I20260928 06:31:51.124624 2453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:31:51.129449 2453 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:31:51.129966 2453 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:31:51.132671 2453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: Generated new TSK 0
I20260928 06:31:51.132830 2453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:31:51.165928 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:51.249305 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:51.249586 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:51.249653 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:51.251017 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:51.251068 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:51.251097 2465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:31:51.252651 2465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:51.252959 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:51.253250 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:51.258706 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:51.258697 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.258770 2465 server_base.cc:1034] running on GCE node
W20260928 06:31:51.258697 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.259227 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:51.259527 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:51.260701 2465 hybrid_clock.cc:648] HybridClock initialized: now 1790577111260696 us; error 38 us; skew 500 ppm
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:51.263603 2465 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:31:51.264076 2465 webserver.cc:460] Webserver started at http://127.24.11.129:45051/ using document root <none> and password file <none>
I20260928 06:31:51.264309 2465 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:51.264379 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:51.264533 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:51.265686 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "5ec904556dfa4c359a68f0ae2c50e274"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "88fc6382b14a9ffd332375cfb932f3c4"
server_key_iv: "e7a940a59eda49d0901c30e259905a82"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.266057 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "5ec904556dfa4c359a68f0ae2c50e274"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "88fc6382b14a9ffd332375cfb932f3c4"
server_key_iv: "e7a940a59eda49d0901c30e259905a82"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.267511 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:31:51.268428 2480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.268622 2465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:51.268725 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "5ec904556dfa4c359a68f0ae2c50e274"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "88fc6382b14a9ffd332375cfb932f3c4"
server_key_iv: "e7a940a59eda49d0901c30e259905a82"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.268848 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:51.290859 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:51.293104 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:51.293300 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:51.293625 2465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:51.294011 2465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:51.294071 2465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.294126 2465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:51.294153 2465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.300201 2465 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:36135
I20260928 06:31:51.300242 2593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:36135 every 8 connection(s)
I20260928 06:31:51.300599 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:31:51.301079 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2465
I20260928 06:31:51.301160 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:31:51.301371 24622 external_mini_cluster.cc:1428] Setting key a2d649a89b60b5d719095fe59318d9ee
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
I20260928 06:31:51.312472 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.301557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:43657 (local address 127.24.11.190:44197)
0928 06:31:51.301608 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.301610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.301893 (+ 283us) server_negotiation.cc:407] Connection header received
0928 06:31:51.302665 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.302666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.302676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.302736 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:51.304055 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.304568 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.306304 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.306513 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.306578 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.309297 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:51.309299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:51.309299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:51.309306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:51.310890 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.311288 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.311289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.311289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.311342 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.311804 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.311807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.311807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.311961 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:51.312021 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.312319 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.312400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:31:51.312790 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.301524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44197 (local address 127.24.11.129:43657)
0928 06:31:51.301727 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.301775 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.302458 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.302841 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.302846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.303700 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:51.303937 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.303943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.304664 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.304665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.306192 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.306196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.306609 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.306611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.306772 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.307344 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:51.307356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:51.309187 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:51.310986 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.310992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.310994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.311217 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.311510 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.311511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.311511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.311725 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.312051 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:51.312058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:51.312189 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.312291 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.312431 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
I20260928 06:31:51.313582 2594 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44197
I20260928 06:31:51.313705 2594 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:51.313938 2594 heartbeater.cc:511] Master 127.24.11.190:44197 requested a full tablet report, sending...
I20260928 06:31:51.314569 2403 ts_manager.cc:194] Registered new tserver with Master: 5ec904556dfa4c359a68f0ae2c50e274 (127.24.11.129:36135)
I20260928 06:31:51.315399 2403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:43657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:31:51.329118 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:51.412734 2599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:51.412954 2599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:51.412978 2599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:51.414211 2599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:51.414256 2599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:51.414283 2599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:31:51.415690 2599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:51.415879 2599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:51.416052 2599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:51.421052 2605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:51.421036 2604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.421197 2599 server_base.cc:1034] running on GCE node
W20260928 06:31:51.421041 2607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.421509 2599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:51.421819 2599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:51.423015 2599 hybrid_clock.cc:648] HybridClock initialized: now 1790577111423005 us; error 30 us; skew 500 ppm
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:51.425747 2599 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:31:51.426200 2599 webserver.cc:460] Webserver started at http://127.24.11.130:38257/ using document root <none> and password file <none>
I20260928 06:31:51.426426 2599 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:51.426496 2599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:51.426664 2599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:51.427757 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "f39e4d5a08d14565aad3a6f707830e27"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "537c8dd844025fe147e861624439c618"
server_key_iv: "02fd382d97bcf8e8f7d161bb86a0d7b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.428128 2599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "f39e4d5a08d14565aad3a6f707830e27"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "537c8dd844025fe147e861624439c618"
server_key_iv: "02fd382d97bcf8e8f7d161bb86a0d7b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.429622 2599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:31:51.430385 2614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.430555 2599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:31:51.430657 2599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f39e4d5a08d14565aad3a6f707830e27"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "537c8dd844025fe147e861624439c618"
server_key_iv: "02fd382d97bcf8e8f7d161bb86a0d7b4"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.430739 2599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:51.452144 2599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:51.454706 2599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:51.454897 2599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:51.455155 2599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:51.455472 2599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:51.455526 2599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.455559 2599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:51.455601 2599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.461326 2599 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:40251
I20260928 06:31:51.461431 2727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:40251 every 8 connection(s)
I20260928 06:31:51.461766 2599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:31:51.464311 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2599
I20260928 06:31:51.464424 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:31:51.464640 24622 external_mini_cluster.cc:1428] Setting key 7956a7f26e2875cb6dc24b486e13ec32
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:51.472287 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.462559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:42219 (local address 127.24.11.190:44197)
0928 06:31:51.462618 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.462621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.462828 (+ 207us) server_negotiation.cc:407] Connection header received
0928 06:31:51.463289 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.463291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.463301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.463397 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:51.464597 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.465252 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.466309 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.466632 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.466705 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.468938 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:51.468939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:51.468940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:51.468948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:51.470741 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.471139 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.471141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.471141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.471214 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.471517 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.471518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.471519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.471700 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:51.471774 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.472077 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.472161 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:31:51.472328 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.462530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44197 (local address 127.24.11.130:42219)
0928 06:31:51.462684 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.462724 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.463201 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.463535 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.463540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.464192 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:51.464490 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.464496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.465335 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.465336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.466204 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.466207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.466719 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.466720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.466832 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.467378 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:51.467387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:51.468831 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:51.470840 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.470845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.470848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.471062 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.471328 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.471329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.471329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.471443 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.471769 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:51.471774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:51.471921 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.472022 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.472157 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260928 06:31:51.472900 2728 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44197
I20260928 06:31:51.473004 2728 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:51.473181 2728 heartbeater.cc:511] Master 127.24.11.190:44197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
I20260928 06:31:51.473635 2403 ts_manager.cc:194] Registered new tserver with Master: f39e4d5a08d14565aad3a6f707830e27 (127.24.11.130:40251)
I20260928 06:31:51.474017 2403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:42219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:31:51.492751 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:31:51.573144 2733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:31:51.573371 2733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:31:51.573396 2733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:31:51.574635 2733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:31:51.574680 2733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:31:51.574709 2733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:31:51.576189 2733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:31:51.576440 2733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:31:51.576663 2733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:31:51.581925 2738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:51.581934 2741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:31:51.581934 2739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:31:51.581951 2733 server_base.cc:1034] running on GCE node
I20260928 06:31:51.582382 2733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:31:51.582646 2733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:31:51.583906 2733 hybrid_clock.cc:648] HybridClock initialized: now 1790577111583898 us; error 36 us; skew 500 ppm
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:31:51.586655 2733 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:31:51.587101 2733 webserver.cc:460] Webserver started at http://127.24.11.131:46037/ using document root <none> and password file <none>
I20260928 06:31:51.587335 2733 fs_manager.cc:359] Metadata directory not provided
I20260928 06:31:51.587381 2733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:31:51.587527 2733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:31:51.588595 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "de5ea7511bec4e08b16e674ecd7319ab"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "304f3daf6fad0fac32d44e12214e1367"
server_key_iv: "a139c02859ec6c86edebb63a7a9b262f"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.588966 2733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "de5ea7511bec4e08b16e674ecd7319ab"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "304f3daf6fad0fac32d44e12214e1367"
server_key_iv: "a139c02859ec6c86edebb63a7a9b262f"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.590468 2733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:31:51.591338 2748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.591522 2733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:31:51.591629 2733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "de5ea7511bec4e08b16e674ecd7319ab"
format_stamp: "Formatted at 2026-09-28 06:31:51 on dist-test-slave-38pg"
server_key: "304f3daf6fad0fac32d44e12214e1367"
server_key_iv: "a139c02859ec6c86edebb63a7a9b262f"
server_key_version: "encryptionkey@0"
I20260928 06:31:51.591706 2733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:31:51.613209 2733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:31:51.615685 2733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:31:51.615870 2733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:31:51.616122 2733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:31:51.616428 2733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:31:51.616483 2733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.616529 2733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:31:51.616554 2733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:31:51.622884 2733 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:35029
I20260928 06:31:51.622917 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:35029 every 8 connection(s)
I20260928 06:31:51.623253 2733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:31:51.627823 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 2733
I20260928 06:31:51.627944 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:31:51.628140 24622 external_mini_cluster.cc:1428] Setting key 1a6517854587258618fe64380b64394d
Sep 28 06:31:51 dist-test-slave-38pg krb5kdc[2358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577111, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:31:51.632572 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.624043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54237 (local address 127.24.11.190:44197)
0928 06:31:51.624105 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.624108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.624308 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:31:51.625236 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.625237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.625246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.625319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:51.626000 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.626494 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.627482 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.627662 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.627716 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.630122 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:51.630123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:51.630123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:51.630131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:51.631316 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.631674 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.631675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.631676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.631726 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.631966 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.631968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.631968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.632101 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:51.632169 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.632415 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.632484 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:31:51.632822 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.624038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44197 (local address 127.24.11.131:54237)
0928 06:31:51.624180 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.624223 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.625147 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.625381 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.625385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.625707 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:31:51.625911 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.625916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.626579 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.626579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.627404 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.627406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.627744 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.627745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.627945 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.628569 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:31:51.628578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:31:51.630035 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:31:51.631388 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.631394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.631396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.631610 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.631783 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:31:51.631784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:31:51.631784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:31:51.631882 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:31:51.632169 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:31:51.632173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:31:51.632292 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.632379 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.632529 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260928 06:31:51.633390 2862 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44197
I20260928 06:31:51.633519 2862 heartbeater.cc:464] Registering TS with master...
I20260928 06:31:51.633714 2862 heartbeater.cc:511] Master 127.24.11.190:44197 requested a full tablet report, sending...
I20260928 06:31:51.634092 2403 ts_manager.cc:194] Registered new tserver with Master: de5ea7511bec4e08b16e674ecd7319ab (127.24.11.131:35029)
I20260928 06:31:51.634479 2403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:54237
I20260928 06:31:51.639967 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:31:51.647575 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.641346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.24.11.190:44197)
0928 06:31:51.641404 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.641407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.641522 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:31:51.641643 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.641644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.641652 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.641694 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:31:51.642249 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.642722 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.643404 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.643583 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.643639 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.644798 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:31:51.644800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:31:51.644800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:31:51.644806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:31:51.646406 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.646818 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.646820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.646820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.646879 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:31:51.647106 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:31:51.647107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:31:51.647107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:31:51.647264 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:31:51.647351 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.647449 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.647511 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:31:51.649372 2403 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:31:51.651036 2403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:31:51.658208 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.24.11.131:35029)
0928 06:31:51.654017 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.654022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.654035 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:31:51.654106 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.654108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.654124 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.654213 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:51.654562 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655746 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657217 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.657779 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657885 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.657950 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.658080 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.658120 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260928 06:31:51.658196 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35029 (local address 127.0.0.1:52630)
0928 06:31:51.653752 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.653822 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.653866 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.654362 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.654362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.654363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:51.654464 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.654466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655849 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.655850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.657133 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.657134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.657887 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.657967 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.657981 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.658056 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.658114 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":26,"thread_start_us":89,"threads_started":1}
I20260928 06:31:51.658370 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35850 (local address 127.24.11.129:36135)
0928 06:31:51.654266 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.654271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.654503 (+ 232us) server_negotiation.cc:407] Connection header received
0928 06:31:51.654571 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.654573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.654587 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.654679 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:51.654799 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655596 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657181 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.657886 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657943 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.657990 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.658220 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.658266 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":83,"threads_started":1}
I20260928 06:31:51.658437 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40251 (local address 127.0.0.1:40548)
0928 06:31:51.653748 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.653790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.653866 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.654353 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.654357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.654360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:51.654537 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.654540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655862 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.655863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.656967 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.656969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.658002 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.658003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.658071 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.658075 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.658126 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.658175 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":88,"thread_start_us":53,"threads_started":1}
I20260928 06:31:51.658571 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36135 (local address 127.0.0.1:35850)
0928 06:31:51.654445 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.654454 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.654507 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.654675 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.654676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.654677 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:51.654743 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.654745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655698 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.655700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.657082 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.657084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.658018 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.658018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.658073 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.658076 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.658126 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.658175 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":37,"thread_start_us":113,"threads_started":1}
I20260928 06:31:51.658905 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.653798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40548 (local address 127.24.11.130:40251)
0928 06:31:51.654203 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.654207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.654216 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:31:51.654252 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.654254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.654269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.654350 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:51.654625 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.655735 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657059 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.657829 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.657907 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.657965 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.658252 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.658812 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":79,"threads_started":1}
I20260928 06:31:51.659245 2796 tablet_service.cc:1467] Processing CreateTablet for tablet 375bdad5a6174f71a4f2657918ce630b (DEFAULT_TABLE table=test-table [id=b9de4d1489bf42c6adac2ab1ec021d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:51.659245 2528 tablet_service.cc:1467] Processing CreateTablet for tablet 375bdad5a6174f71a4f2657918ce630b (DEFAULT_TABLE table=test-table [id=b9de4d1489bf42c6adac2ab1ec021d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:51.659626 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375bdad5a6174f71a4f2657918ce630b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:51.659626 2528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375bdad5a6174f71a4f2657918ce630b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:51.662151 2662 tablet_service.cc:1467] Processing CreateTablet for tablet 375bdad5a6174f71a4f2657918ce630b (DEFAULT_TABLE table=test-table [id=b9de4d1489bf42c6adac2ab1ec021d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:31:51.662508 2662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 375bdad5a6174f71a4f2657918ce630b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:31:51.663125 2879 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Bootstrap starting.
I20260928 06:31:51.663889 2879 tablet_bootstrap.cc:654] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:51.664052 2880 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Bootstrap starting.
I20260928 06:31:51.664587 2879 log.cc:824] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:51.664692 2880 tablet_bootstrap.cc:654] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:51.665243 2880 log.cc:824] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:51.665778 2883 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Bootstrap starting.
I20260928 06:31:51.666095 2880 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: No bootstrap required, opened a new log
I20260928 06:31:51.666150 2880 ts_tablet_manager.cc:1397] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:31:51.666368 2883 tablet_bootstrap.cc:654] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Neither blocks nor log segments found. Creating new log.
I20260928 06:31:51.666810 2883 log.cc:824] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Log is configured to *not* fsync() on all Append() calls
I20260928 06:31:51.667611 2883 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: No bootstrap required, opened a new log
I20260928 06:31:51.667711 2883 ts_tablet_manager.cc:1397] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:31:51.667927 2880 raft_consensus.cc:348] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.668066 2880 raft_consensus.cc:374] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:51.668088 2880 raft_consensus.cc:729] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec904556dfa4c359a68f0ae2c50e274, State: Initialized, Role: FOLLOWER
I20260928 06:31:51.668226 2880 consensus_queue.cc:260] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.668500 2880 ts_tablet_manager.cc:1428] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:31:51.668565 2594 heartbeater.cc:503] Master 127.24.11.190:44197 was elected leader, sending a full tablet report...
I20260928 06:31:51.669001 2879 tablet_bootstrap.cc:492] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: No bootstrap required, opened a new log
I20260928 06:31:51.669077 2879 ts_tablet_manager.cc:1397] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 06:31:51.669826 2883 raft_consensus.cc:348] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.669998 2883 raft_consensus.cc:374] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:51.670037 2883 raft_consensus.cc:729] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39e4d5a08d14565aad3a6f707830e27, State: Initialized, Role: FOLLOWER
I20260928 06:31:51.670156 2883 consensus_queue.cc:260] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.670452 2883 ts_tablet_manager.cc:1428] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:31:51.670470 2728 heartbeater.cc:503] Master 127.24.11.190:44197 was elected leader, sending a full tablet report...
I20260928 06:31:51.671205 2879 raft_consensus.cc:348] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.671394 2879 raft_consensus.cc:374] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:31:51.671446 2879 raft_consensus.cc:729] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5ea7511bec4e08b16e674ecd7319ab, State: Initialized, Role: FOLLOWER
I20260928 06:31:51.671562 2879 consensus_queue.cc:260] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.671808 2879 ts_tablet_manager.cc:1428] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:31:51.671842 2862 heartbeater.cc:503] Master 127.24.11.190:44197 was elected leader, sending a full tablet report...
W20260928 06:31:51.712379 2729 tablet.cc:2369] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:51.801396 2595 tablet.cc:2369] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:31:51.873874 2863 tablet.cc:2369] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:31:51.881436 2887 raft_consensus.cc:482] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:31:51.881619 2887 raft_consensus.cc:504] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.882081 2887 leader_election.cc:288] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f39e4d5a08d14565aad3a6f707830e27 (127.24.11.130:40251), 5ec904556dfa4c359a68f0ae2c50e274 (127.24.11.129:36135)
I20260928 06:31:51.886257 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.882300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:56143 (local address 127.24.11.129:36135)
0928 06:31:51.882345 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.882347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.882546 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:31:51.882587 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.882588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.882601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.882649 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:51.882800 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.883562 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885227 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.885830 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885918 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.885929 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.886145 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.886186 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:31:51.886257 2876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.882382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40881 (local address 127.24.11.130:40251)
0928 06:31:51.882449 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.882451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.882459 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:31:51.882491 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.882492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.882501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.882573 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:31:51.882792 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.883562 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885227 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.885877 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885945 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.885953 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.886145 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.886185 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:31:51.886278 2864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.882237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40251 (local address 127.24.11.131:40881)
0928 06:31:51.882299 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.882313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.882372 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.882596 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.882597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.882598 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:51.882691 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.882692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.883673 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.883674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.885086 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.885089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.885953 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.886052 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.886054 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.886125 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.886179 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":113}
I20260928 06:31:51.886284 2888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.882252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36135 (local address 127.24.11.131:56143)
0928 06:31:51.882455 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.882466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:31:51.882534 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:31:51.882663 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:31:51.882664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:31:51.882665 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:31:51.882727 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.882728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.883673 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.883674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.885089 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:31:51.885091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.885953 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.885954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:31:51.886049 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.886054 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:31:51.886125 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:31:51.886179 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":91,"thread_start_us":101,"threads_started":1}
I20260928 06:31:51.886672 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375bdad5a6174f71a4f2657918ce630b" candidate_uuid: "de5ea7511bec4e08b16e674ecd7319ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39e4d5a08d14565aad3a6f707830e27" is_pre_election: true
I20260928 06:31:51.886852 2682 raft_consensus.cc:2454] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5ea7511bec4e08b16e674ecd7319ab in term 0.
I20260928 06:31:51.886900 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375bdad5a6174f71a4f2657918ce630b" candidate_uuid: "de5ea7511bec4e08b16e674ecd7319ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec904556dfa4c359a68f0ae2c50e274" is_pre_election: true
I20260928 06:31:51.887053 2548 raft_consensus.cc:2454] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de5ea7511bec4e08b16e674ecd7319ab in term 0.
I20260928 06:31:51.887109 2750 leader_election.cc:302] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de5ea7511bec4e08b16e674ecd7319ab, f39e4d5a08d14565aad3a6f707830e27; no voters:
I20260928 06:31:51.887302 2887 raft_consensus.cc:2781] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:31:51.887365 2887 raft_consensus.cc:482] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:31:51.887394 2887 raft_consensus.cc:3037] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:51.888100 2887 raft_consensus.cc:504] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.888249 2887 leader_election.cc:288] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [CANDIDATE]: Term 1 election: Requested vote from peers f39e4d5a08d14565aad3a6f707830e27 (127.24.11.130:40251), 5ec904556dfa4c359a68f0ae2c50e274 (127.24.11.129:36135)
I20260928 06:31:51.888399 2682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375bdad5a6174f71a4f2657918ce630b" candidate_uuid: "de5ea7511bec4e08b16e674ecd7319ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39e4d5a08d14565aad3a6f707830e27"
I20260928 06:31:51.888407 2548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "375bdad5a6174f71a4f2657918ce630b" candidate_uuid: "de5ea7511bec4e08b16e674ecd7319ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec904556dfa4c359a68f0ae2c50e274"
I20260928 06:31:51.888469 2682 raft_consensus.cc:3037] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:51.888497 2548 raft_consensus.cc:3037] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:31:51.889228 2548 raft_consensus.cc:2454] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5ea7511bec4e08b16e674ecd7319ab in term 1.
I20260928 06:31:51.889268 2682 raft_consensus.cc:2454] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de5ea7511bec4e08b16e674ecd7319ab in term 1.
I20260928 06:31:51.889431 2752 leader_election.cc:302] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec904556dfa4c359a68f0ae2c50e274, de5ea7511bec4e08b16e674ecd7319ab; no voters:
I20260928 06:31:51.889518 2887 raft_consensus.cc:2781] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:31:51.889701 2887 raft_consensus.cc:686] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 1 LEADER]: Becoming Leader. State: Replica: de5ea7511bec4e08b16e674ecd7319ab, State: Running, Role: LEADER
I20260928 06:31:51.889806 2887 consensus_queue.cc:237] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } }
I20260928 06:31:51.890622 2402 catalog_manager.cc:5495] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab reported cstate change: term changed from 0 to 1, leader changed from <none> to de5ea7511bec4e08b16e674ecd7319ab (127.24.11.131). New cstate: current_term: 1 leader_uuid: "de5ea7511bec4e08b16e674ecd7319ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de5ea7511bec4e08b16e674ecd7319ab" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 35029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 } health_report { overall_health: UNKNOWN } } }
I20260928 06:31:51.940981 2877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.938416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.24.11.131:35029)
0928 06:31:51.938481 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:31:51.938483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:31:51.938493 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:31:51.938549 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:31:51.938550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:31:51.938569 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:31:51.938657 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:31:51.938818 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.939317 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.940110 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:31:51.940321 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:31:51.940392 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:31:51.940567 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:31:51.940678 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:31:51.940784 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:31:51.940824 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:31:51.940847 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:31:51.942880 2682 raft_consensus.cc:1261] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 1 FOLLOWER]: Refusing update from remote peer de5ea7511bec4e08b16e674ecd7319ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:51.942937 2548 raft_consensus.cc:1261] T 375bdad5a6174f71a4f2657918ce630b P 5ec904556dfa4c359a68f0ae2c50e274 [term 1 FOLLOWER]: Refusing update from remote peer de5ea7511bec4e08b16e674ecd7319ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:31:51.943223 2887 consensus_queue.cc:1035] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39e4d5a08d14565aad3a6f707830e27" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:51.943408 2889 consensus_queue.cc:1035] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec904556dfa4c359a68f0ae2c50e274" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:31:51.945798 2892 mvcc.cc:205] Tried to move back new op lower bound from 7334203850515312640 to 7334203850301202432. Current Snapshot: MvccSnapshot[applied={T|T < 7334203850515312640}]
I20260928 06:31:51.948565 2402 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:51.948666 2402 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:31:51.949499 2402 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0daf707007254264b758567b34fb99da: Sending DeleteTablet for 3 replicas of tablet 375bdad5a6174f71a4f2657918ce630b
I20260928 06:31:51.949857 2662 tablet_service.cc:1514] Processing DeleteTablet for tablet 375bdad5a6174f71a4f2657918ce630b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:51 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:40548
I20260928 06:31:51.949858 2528 tablet_service.cc:1514] Processing DeleteTablet for tablet 375bdad5a6174f71a4f2657918ce630b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:51 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:35850
I20260928 06:31:51.949913 2796 tablet_service.cc:1514] Processing DeleteTablet for tablet 375bdad5a6174f71a4f2657918ce630b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:31:51 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:52630
I20260928 06:31:51.950112 2901 tablet_replica.cc:330] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: stopping tablet replica
I20260928 06:31:51.950282 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2465
I20260928 06:31:51.950376 2901 raft_consensus.cc:2229] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:31:51.954442 2901 raft_consensus.cc:2258] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:51.956571 2901 ts_tablet_manager.cc:1905] T 375bdad5a6174f71a4f2657918ce630b P f39e4d5a08d14565aad3a6f707830e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:31:51.957707 2903 tablet_replica.cc:330] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab: stopping tablet replica
W20260928 06:31:51.957801 2390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:36135 (error 108)
I20260928 06:31:51.957849 2903 raft_consensus.cc:2229] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:31:51.957974 2903 raft_consensus.cc:2258] T 375bdad5a6174f71a4f2657918ce630b P de5ea7511bec4e08b16e674ecd7319ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:31:51.958056 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2599
I20260928 06:31:51.958901 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.958782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36135 (local address 127.0.0.1:35862)
0928 06:31:51.958828 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.958865 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:31:51.959079 2390 catalog_manager.cc:4584] TS 5ec904556dfa4c359a68f0ae2c50e274 (127.24.11.129:36135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 375bdad5a6174f71a4f2657918ce630b: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36135: connect: Connection refused (error 111)
I20260928 06:31:51.963001 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2733
I20260928 06:31:51.963063 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.962953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40251 (local address 127.0.0.1:40558)
0928 06:31:51.962997 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.963025 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:40251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:31:51.967820 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 2371
I20260928 06:31:51.967901 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:31:51.967768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:35029 (local address 127.0.0.1:52648)
0928 06:31:51.967819 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:31:51.967853 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.131:35029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
2026-09-28T06:31:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2135 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:31:52 dist-test-slave-38pg krb5kdc[2907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:31:52 dist-test-slave-38pg krb5kdc[2907](info): set up 2 sockets
Sep 28 06:31:52 dist-test-slave-38pg krb5kdc[2907](info): commencing operation
krb5kdc: starting...
W20260928 06:31:53.011722 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:31:53 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:31:53Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangeradmin_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangeradmin_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangerlookup_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangerlookup_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
I20260928 06:31:53.180653 24622 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwumdS4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/postgres -l logfile start
2026-09-28 06:31:55.893 UTC [2937] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:31:55.893 UTC [2937] LOG: listening on IPv4 address "127.24.11.148", port 33087
2026-09-28 06:31:55.896 UTC [2937] LOG: listening on Unix socket "/tmp/.s.PGSQL.33087"
2026-09-28 06:31:55.917 UTC [2940] LOG: database system was shut down at 2026-09-28 06:31:54 UTC
2026-09-28 06:31:55.921 UTC [2937] LOG: database system is ready to accept connections
I20260928 06:31:56.891716 24622 mini_postgres.cc:96] Postgres bound to 33087
2026-09-28 06:31:56.899 UTC [2948] FATAL: database "slave" does not exist
127.24.11.148:33087 - accepting connections
I20260928 06:31:56.899782 24622 mini_ranger.cc:161] Starting Ranger...
I20260928 06:31:56.916364 24622 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:31:57.474463 24622 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:31:57.474556 24622 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin
2026-09-28 06:31:57,994 [I] DB FLAVOR :POSTGRES
2026-09-28 06:31:57,995 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:31:57,995 [I] Checking connection..
2026-09-28 06:31:57,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:31:58,282 [I] Checking connection passed.
2026-09-28 06:31:58,282 [I] --------- Verifying version history table ---------
2026-09-28 06:31:58,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:31:58,549 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:31:58,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:31:58,811 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:31:58,811 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:31:58,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:31:59.080 UTC [3012] WARNING: there is no transaction in progress
2026-09-28 06:31:59,087 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:31:59,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:31:59,338 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:31:59,338 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:31:59,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:31:59,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:31:59,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-38pg.c.gcp-upstream.internal','N') ;"
2026-09-28 06:32:00,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:32:00,360 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:32:00,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:32:00,634 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:32:00,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:32:00,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:32:01,168 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:32:01,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:32:02.246 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.263 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.280 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.297 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.314 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.337 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.398 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.405 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.422 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.434 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.445 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.460 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.472 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.480 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.489 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:02.497 UTC [3120] WARNING: there is no transaction in progress
2026-09-28 06:32:03,233 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:32:03,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:32:03,500 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:32:03,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:32:03,758 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:32:03,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:32:04,027 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:32:04,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:32:04,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:32:04,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-38pg.c.gcp-upstream.internal';"
2026-09-28 06:32:04,776 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:32:04,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:32:05,043 [I] DB_PATCHES have already been applied
I20260928 06:32:05.049397 24622 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwumdS4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:32:05.049475 24622 mini_ranger.cc:193] Using host: 127.24.11.148
I20260928 06:32:05.052511 24622 mini_ranger.cc:227] Ranger admin URL: http://127.24.11.148:51463
Sep 28, 2026 6:32:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:32:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:32:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:32:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:32:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:32:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:32:07.330250 24622 mini_ranger.cc:160] Time spent starting Ranger: real 10.430s user 0.000s sys 0.011s
Sep 28, 2026 6:32:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:32:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 6:32:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:32:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:32:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:32:30.031934 24622 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangerkms_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/rangerkms_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.11.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.11.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:32:30 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577150, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:32:30.115159 24622 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:32:30.124768 24622 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:32:31.020186 24622 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:32:31.020289 24622 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms
I20260928 06:32:31.020309 24622 mini_ranger_kms.cc:227] Using postgres at 127.24.11.148:33087
2026-09-28 06:32:31,612 [I] DB FLAVOR :POSTGRES
2026-09-28 06:32:31,612 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:32:31,612 [I] Checking connection
2026-09-28 06:32:31,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:32:31,887 [I] connection success
2026-09-28 06:32:31,888 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:32:31,888 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:32:31,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:32:32,170 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:32:32,170 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:32:32,170 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:32:32,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:33087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:32:32,431 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:32:32.943153 24622 mini_ranger_kms.cc:310] Created kms service
I20260928 06:32:33.164202 24622 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:32:33.205467 24622 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:32:33 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577150, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:32:33.590216 24622 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:32:33.590340 24622 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:32:33.590375 24622 mini_ranger_kms.cc:240] Using host: 127.24.11.148
I20260928 06:32:33.594365 24622 mini_ranger_kms.cc:276] Ranger KMS PID: 3302
I20260928 06:32:33.594483 24622 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.11.148:35115
Sep 28, 2026 6:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:32:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:32:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 06:32:35 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577155, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:35.859933 24622 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.745s user 0.004s sys 0.233s
I20260928 06:32:35.860045 24622 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:32:35.860132 24622 mini_ranger_kms.cc:401] 127.24.11.148:35115/kms/v1/keys
Sep 28 06:32:36 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577155, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
Sep 28 06:32:36 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577155, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
Sep 28 06:32:38 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:32:39.028513 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:46683
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:46683
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:35115/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:39.129837 3352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:39.130049 3352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:39.130074 3352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:39.131177 3352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:32:39.131212 3352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:39.131227 3352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:39.131239 3352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:32:39.131251 3352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:32:39.132740 3352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.11.148:35115/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:46683
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:46683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:32:39.132922 3352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:39.133087 3352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:39.138381 3357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:39.138368 3358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:39.138362 3352 server_base.cc:1034] running on GCE node
W20260928 06:32:39.138391 3360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:39.138911 3352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:39.139209 3352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:39.140360 3352 hybrid_clock.cc:648] HybridClock initialized: now 1790577159140347 us; error 30 us; skew 500 ppm
Sep 28 06:32:39 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:39.142987 3352 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:32:39.143402 3352 webserver.cc:460] Webserver started at http://127.24.11.190:37585/ using document root <none> and password file <none>
I20260928 06:32:39.143649 3352 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:39.143707 3352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:39.143853 3352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:32:39 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577159, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:32:39.205859 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "53193d6fb8384e44aee2f3a92b8aba1a"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-38pg"
server_key: "c1c52d09a876d73a2a7ae3fe6eb6ecee"
server_key_iv: "90eacadf74d09f554bc3e6b33e5ace6e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:39.206377 3352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "53193d6fb8384e44aee2f3a92b8aba1a"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-38pg"
server_key: "c1c52d09a876d73a2a7ae3fe6eb6ecee"
server_key_iv: "90eacadf74d09f554bc3e6b33e5ace6e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:39.207836 3352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:32:39.220024 3367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:39.220238 3352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:32:39.220348 3352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "53193d6fb8384e44aee2f3a92b8aba1a"
format_stamp: "Formatted at 2026-09-28 06:32:39 on dist-test-slave-38pg"
server_key: "c1c52d09a876d73a2a7ae3fe6eb6ecee"
server_key_iv: "90eacadf74d09f554bc3e6b33e5ace6e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:39.220471 3352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:39.255411 3352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:39.257467 3352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:39.257694 3352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:39.263055 3352 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:46683
I20260928 06:32:39.263077 3419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:46683 every 8 connection(s)
I20260928 06:32:39.263480 3352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:32:39.264274 3420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:39.265676 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 3352
I20260928 06:32:39.265800 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:32:39.266023 24622 external_mini_cluster.cc:1428] Setting key ebef0723825cfd100050c9d4449cc6c4
I20260928 06:32:39.267627 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: Bootstrap starting.
I20260928 06:32:39.268435 3420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:39.268762 3420 log.cc:824] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:39.269606 3420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: No bootstrap required, opened a new log
I20260928 06:32:39.272176 3420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } }
I20260928 06:32:39.272372 3420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:39.272408 3420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53193d6fb8384e44aee2f3a92b8aba1a, State: Initialized, Role: FOLLOWER
I20260928 06:32:39.272567 3420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } }
I20260928 06:32:39.272643 3420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:32:39.272673 3420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:32:39.272735 3420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 0 FOLLOWER]: Advancing to term 1
Sep 28 06:32:39 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:32:39.273591 3420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } }
I20260928 06:32:39.273707 3420 leader_election.cc:302] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53193d6fb8384e44aee2f3a92b8aba1a; no voters:
I20260928 06:32:39.273914 3420 leader_election.cc:288] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:32:39.273964 3425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:39.274143 3425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [term 1 LEADER]: Becoming Leader. State: Replica: 53193d6fb8384e44aee2f3a92b8aba1a, State: Running, Role: LEADER
I20260928 06:32:39.274220 3420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:32:39.274559 3425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } }
I20260928 06:32:39.275079 3425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53193d6fb8384e44aee2f3a92b8aba1a. Latest consensus state: current_term: 1 leader_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } } }
I20260928 06:32:39.275247 3425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:39.276186 3426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53193d6fb8384e44aee2f3a92b8aba1a" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46683 } } }
I20260928 06:32:39.276297 3426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a [sys.catalog]: This master's current role is: LEADER
I20260928 06:32:39.276561 3420 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:32:39.276930 3437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:32:39.279780 3437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:32:39.280289 3423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:39.266571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.24.11.190:46683)
0928 06:32:39.266802 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:39.266810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:39.266833 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:32:39.267261 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:39.267267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:39.267314 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:39.267483 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:39.268189 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.269336 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.270221 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:39.270672 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:39.270796 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:39.273308 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:39.273312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:39.273312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:39.273324 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:39.275467 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.275887 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.275891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.275891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.276000 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:39.276249 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:39.276250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:39.276250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:39.276573 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:39.276732 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:39.278985 (+ 2253us) server_negotiation.cc:299] Negotiation successful
0928 06:32:39.279880 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260928 06:32:39.282394 3437 catalog_manager.cc:1347] Generated new cluster ID: f9c41c6f7c5b4648a32aad0f11320cdb
I20260928 06:32:39.282454 3437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:32:39.314910 3437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:32:39.315523 3437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:32:39.323184 3437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: Generated new TSK 0
I20260928 06:32:39.323386 3437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:32:40.041720 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:35115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:40.132700 3465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.132942 3465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.132983 3465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.134258 3465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.134322 3465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.134374 3465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:32:40.135970 3465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.11.148:35115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:32:40.136222 3465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.136457 3465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.141546 3473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.141600 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.141672 3471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.142591 3465 server_base.cc:1034] running on GCE node
I20260928 06:32:40.142812 3465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.143106 3465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.144261 3465 hybrid_clock.cc:648] HybridClock initialized: now 1790577160144250 us; error 28 us; skew 500 ppm
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.146955 3465 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:32:40.147439 3465 webserver.cc:460] Webserver started at http://127.24.11.129:37429/ using document root <none> and password file <none>
I20260928 06:32:40.147666 3465 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.147729 3465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.147871 3465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:32:40.162001 3465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "8f31611e8f6341d684c6f26fbb14962b"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "aec18f1e4ab94cae4f02eb2ad6d132c4"
server_key_iv: "86d65bd917be325c06f39d446572974e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.162568 3465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "8f31611e8f6341d684c6f26fbb14962b"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "aec18f1e4ab94cae4f02eb2ad6d132c4"
server_key_iv: "86d65bd917be325c06f39d446572974e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.164472 3465 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:32:40.178639 3480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.178880 3465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:32:40.178980 3465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "8f31611e8f6341d684c6f26fbb14962b"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "aec18f1e4ab94cae4f02eb2ad6d132c4"
server_key_iv: "86d65bd917be325c06f39d446572974e"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.179093 3465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.194787 3465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.196899 3465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.197059 3465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.197316 3465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.197775 3465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.197829 3465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.197886 3465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.197921 3465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.205502 3465 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:33761
I20260928 06:32:40.206121 3465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:32:40.208374 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 3465
I20260928 06:32:40.208492 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:32:40.208693 24622 external_mini_cluster.cc:1428] Setting key 84eba534609366846528c100fcfb18ee
I20260928 06:32:40.218148 3593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:33761 every 8 connection(s)
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
I20260928 06:32:40.233410 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.218902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46683 (local address 127.24.11.129:41617)
0928 06:32:40.219453 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.219515 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.220301 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.220775 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.220781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.221043 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.221362 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.221371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.222472 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.222473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.223839 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.223842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.224390 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.224391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.224578 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.225219 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:40.225231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.229316 (+ 4085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.231534 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.231544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.231548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.231913 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.232157 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.232158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.232158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.232309 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.232649 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.232656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.232838 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.232961 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.233141 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":102,"threads_started":1}
I20260928 06:32:40.233886 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.219248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:41617 (local address 127.24.11.190:46683)
0928 06:32:40.220513 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.220515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.220524 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:32:40.220554 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.220554 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.220569 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.220636 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.221492 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.222336 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.223953 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.224237 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.224307 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.229467 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.229469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.229469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.229479 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.231422 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.232009 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.232010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.232010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.232080 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.232401 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.232403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.232403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.232555 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.232619 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.233702 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.233795 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":94,"threads_started":1}
I20260928 06:32:40.234258 3594 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46683
I20260928 06:32:40.234385 3594 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.234588 3594 heartbeater.cc:511] Master 127.24.11.190:46683 requested a full tablet report, sending...
I20260928 06:32:40.235131 3384 ts_manager.cc:194] Registered new tserver with Master: 8f31611e8f6341d684c6f26fbb14962b (127.24.11.129:33761)
I20260928 06:32:40.235996 3384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:41617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:32:40.250856 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:35115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:40.374316 3602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.374624 3602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.374670 3602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.376588 3602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.376657 3602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.376699 3602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:32:40.379228 3602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.11.148:35115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:32:40.379513 3602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.379889 3602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.386538 3612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.386821 3609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.387042 3602 server_base.cc:1034] running on GCE node
W20260928 06:32:40.386577 3610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.387363 3602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.387658 3602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.388814 3602 hybrid_clock.cc:648] HybridClock initialized: now 1790577160388793 us; error 48 us; skew 500 ppm
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.392279 3602 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:32:40.392926 3602 webserver.cc:460] Webserver started at http://127.24.11.130:42389/ using document root <none> and password file <none>
I20260928 06:32:40.393227 3602 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.393296 3602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.393499 3602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:32:40.408051 3602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "fdf23fb0b4b447d29579a02ea0a499a7"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "f7a54bc7f49561cdbeb12fa7d939ba6b"
server_key_iv: "0a7ccd2aff55ccd12c51212b214b0f62"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.408550 3602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "fdf23fb0b4b447d29579a02ea0a499a7"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "f7a54bc7f49561cdbeb12fa7d939ba6b"
server_key_iv: "0a7ccd2aff55ccd12c51212b214b0f62"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.412678 3602 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.438793 3619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.439090 3602 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 06:32:40.439175 3602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "fdf23fb0b4b447d29579a02ea0a499a7"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "f7a54bc7f49561cdbeb12fa7d939ba6b"
server_key_iv: "0a7ccd2aff55ccd12c51212b214b0f62"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.439284 3602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.464449 3602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.468019 3602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.468250 3602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.468573 3602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.468971 3602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.469054 3602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.469132 3602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.469188 3602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.475956 3602 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33917
I20260928 06:32:40.476460 3602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:32:40.478861 3732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33917 every 8 connection(s)
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:32:40.485301 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 3602
I20260928 06:32:40.485396 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:32:40.485626 24622 external_mini_cluster.cc:1428] Setting key dd8f61eddebf4be7949b058df3139041
I20260928 06:32:40.488710 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.477593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:50861 (local address 127.24.11.190:46683)
0928 06:32:40.477644 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.477646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.478019 (+ 373us) server_negotiation.cc:407] Connection header received
0928 06:32:40.478776 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.478777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.478794 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.478838 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.479699 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.480512 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.482104 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.482444 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.482521 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.485241 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.485243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.485243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.485252 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.487104 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.487598 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.487599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.487600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.487659 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.487954 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.487955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.487955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.488091 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.488135 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.488520 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.488613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:32:40.489216 3735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.477246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46683 (local address 127.24.11.130:50861)
0928 06:32:40.477880 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.477928 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.478642 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.479037 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.479042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.479307 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.479584 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.479591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.480624 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.480625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.482005 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.482008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.482649 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.482650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.482798 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.483440 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:40.483451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.485142 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.487199 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.487206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.487209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.487505 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.487731 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.487731 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.487732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.487874 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.488182 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.488188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.488359 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.488809 (+ 450us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.488989 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":99,"threads_started":1}
I20260928 06:32:40.489840 3733 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46683
I20260928 06:32:40.489936 3733 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.490123 3733 heartbeater.cc:511] Master 127.24.11.190:46683 requested a full tablet report, sending...
I20260928 06:32:40.490527 3384 ts_manager.cc:194] Registered new tserver with Master: fdf23fb0b4b447d29579a02ea0a499a7 (127.24.11.130:33917)
I20260928 06:32:40.491029 3384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:50861
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:32:40.522688 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:35115/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:32:40.649149 3739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:32:40.649453 3739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:32:40.649495 3739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:32:40.651366 3739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:32:40.651433 3739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:32:40.651472 3739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:32:40.653825 3739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.11.148:35115/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:32:40.654106 3739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:32:40.654440 3739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:32:40.660980 3744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.660961 3745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:32:40.661190 3747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:32:40.661810 3739 server_base.cc:1034] running on GCE node
I20260928 06:32:40.662014 3739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:32:40.662276 3739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:32:40.663444 3739 hybrid_clock.cc:648] HybridClock initialized: now 1790577160663424 us; error 35 us; skew 500 ppm
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:32:40.666949 3739 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:32:40.667546 3739 webserver.cc:460] Webserver started at http://127.24.11.131:43749/ using document root <none> and password file <none>
I20260928 06:32:40.667841 3739 fs_manager.cc:359] Metadata directory not provided
I20260928 06:32:40.667972 3739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:32:40.668129 3739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:32:40.683637 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "2f17883342b54a679950e917b643d094"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "15b1c17faabc711caa502ca8bfa8917e"
server_key_iv: "d0ecf4aac50d4e0e53420942ac672153"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.684119 3739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "2f17883342b54a679950e917b643d094"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "15b1c17faabc711caa502ca8bfa8917e"
server_key_iv: "d0ecf4aac50d4e0e53420942ac672153"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.686753 3739 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 06:32:40.702083 3754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.702370 3739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:32:40.702473 3739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "2f17883342b54a679950e917b643d094"
format_stamp: "Formatted at 2026-09-28 06:32:40 on dist-test-slave-38pg"
server_key: "15b1c17faabc711caa502ca8bfa8917e"
server_key_iv: "d0ecf4aac50d4e0e53420942ac672153"
server_key_version: "kuduclusterkey@0"
I20260928 06:32:40.702577 3739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:32:40.744990 3739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:32:40.747702 3739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:32:40.747921 3739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:32:40.748229 3739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:32:40.748687 3739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:32:40.748761 3739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.748801 3739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:32:40.748835 3739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:32:40.758224 3739 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:45053
I20260928 06:32:40.767118 3739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:32:40.758303 3867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:45053 every 8 connection(s)
I20260928 06:32:40.776002 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 3739
I20260928 06:32:40.776124 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:32:40.776342 24622 external_mini_cluster.cc:1428] Setting key 3f9beb5580965b36807a06829582bb54
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:32:40.790248 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.778312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:49945 (local address 127.24.11.190:46683)
0928 06:32:40.778371 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.778374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.779828 (+ 1454us) server_negotiation.cc:407] Connection header received
0928 06:32:40.780671 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.780673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.780686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.780746 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.781374 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.782029 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.783274 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.783488 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.783574 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.785822 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.785824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.785824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.785832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.787318 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.789202 (+ 1884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.789203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.789204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.789276 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.789556 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.789557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.789557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.789684 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.789762 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.790054 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.790143 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:32:40.790450 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.778339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46683 (local address 127.24.11.131:49945)
0928 06:32:40.779537 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.779636 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.780535 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.780831 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.780841 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.781052 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:32:40.781270 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.781274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.782127 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.782128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.783169 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.783171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.783583 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.783584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.783732 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.784332 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:32:40.784342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:32:40.785716 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:32:40.787421 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.787428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.787431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.789089 (+ 1658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.789346 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:32:40.789347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:32:40.789347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:32:40.789478 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:32:40.789753 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:32:40.789760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:32:40.789919 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.790034 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.790192 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":123,"threads_started":1}
I20260928 06:32:40.790920 3868 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46683
I20260928 06:32:40.791019 3868 heartbeater.cc:464] Registering TS with master...
I20260928 06:32:40.791213 3868 heartbeater.cc:511] Master 127.24.11.190:46683 requested a full tablet report, sending...
I20260928 06:32:40.791677 3384 ts_manager.cc:194] Registered new tserver with Master: 2f17883342b54a679950e917b643d094 (127.24.11.131:45053)
I20260928 06:32:40.792114 3384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:49945
I20260928 06:32:40.799080 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:32:40.809669 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.800807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.24.11.190:46683)
0928 06:32:40.800879 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.800881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.801116 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:32:40.801162 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.801163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.801174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.801243 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:32:40.802119 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.802922 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.804018 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.804305 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.804357 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.806201 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:32:40.806203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:32:40.806203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:32:40.806211 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:32:40.808306 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.808748 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.808749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.808750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.808826 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:32:40.809041 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:32:40.809042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:32:40.809043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:32:40.809192 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:32:40.809300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.809456 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.809571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:32:40.811662 3384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:32:40.814155 3384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:32:40.830610 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.817352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45053 (local address 127.0.0.1:60242)
0928 06:32:40.825800 (+ 8448us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.825810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.825852 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.826082 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.826083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.826086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.826190 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.826192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.827842 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.827843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.829188 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.829191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.830284 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.830285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.830386 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.830405 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.830461 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.830513 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8433,"mutex_wait_us":5,"thread_start_us":69,"threads_started":1}
I20260928 06:32:40.830924 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.817054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33917 (local address 127.0.0.1:45046)
0928 06:32:40.825613 (+ 8559us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.825628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.825709 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.826071 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.826075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.826078 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.826182 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.826185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.827690 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.827691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.829398 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.829401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.830637 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.830638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.830719 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.830723 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.830782 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.830841 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8535,"thread_start_us":117,"threads_started":1}
I20260928 06:32:40.831394 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.817984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.24.11.131:45053)
0928 06:32:40.821211 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.821215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.825905 (+ 4690us) server_negotiation.cc:407] Connection header received
0928 06:32:40.825956 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.825959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.825977 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.826276 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.826359 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.827707 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.829322 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.830171 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.831154 (+ 983us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.831246 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.831273 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.831312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3190,"thread_start_us":93,"threads_started":1}
I20260928 06:32:40.832015 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.817693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.24.11.130:33917)
0928 06:32:40.821117 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.821123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.825725 (+ 4602us) server_negotiation.cc:407] Connection header received
0928 06:32:40.825802 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.825806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.825828 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.825939 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.826272 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.827578 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.829573 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.830434 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.830513 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.830597 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.831008 (+ 411us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.831054 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3331,"thread_start_us":179,"threads_started":1}
I20260928 06:32:40.832190 3802 tablet_service.cc:1467] Processing CreateTablet for tablet 892276ec642f4478ba5630aa08822c5f (DEFAULT_TABLE table=test-table [id=33e54e160dde44e28a6067da747caa06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.832581 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892276ec642f4478ba5630aa08822c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.832834 3667 tablet_service.cc:1467] Processing CreateTablet for tablet 892276ec642f4478ba5630aa08822c5f (DEFAULT_TABLE table=test-table [id=33e54e160dde44e28a6067da747caa06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.833163 3667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892276ec642f4478ba5630aa08822c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.836303 3886 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Bootstrap starting.
I20260928 06:32:40.837692 3886 tablet_bootstrap.cc:654] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.838017 3886 log.cc:824] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.838353 3887 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Bootstrap starting.
I20260928 06:32:40.839478 3887 tablet_bootstrap.cc:654] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.839778 3887 log.cc:824] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.840767 3887 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: No bootstrap required, opened a new log
I20260928 06:32:40.840714 3886 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: No bootstrap required, opened a new log
I20260928 06:32:40.840852 3887 ts_tablet_manager.cc:1397] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:32:40.841431 3886 ts_tablet_manager.cc:1397] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 06:32:40.842893 3887 raft_consensus.cc:348] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.843077 3887 raft_consensus.cc:374] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.843113 3887 raft_consensus.cc:729] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf23fb0b4b447d29579a02ea0a499a7, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.843282 3887 consensus_queue.cc:260] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.843704 3887 ts_tablet_manager.cc:1428] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:32:40.843855 3733 heartbeater.cc:503] Master 127.24.11.190:46683 was elected leader, sending a full tablet report...
I20260928 06:32:40.845276 3886 raft_consensus.cc:348] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.845577 3886 raft_consensus.cc:374] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.845675 3886 raft_consensus.cc:729] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f17883342b54a679950e917b643d094, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.845901 3886 consensus_queue.cc:260] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.846271 3886 ts_tablet_manager.cc:1428] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260928 06:32:40.846472 3868 heartbeater.cc:503] Master 127.24.11.190:46683 was elected leader, sending a full tablet report...
I20260928 06:32:40.849321 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.820430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.24.11.129:33761)
0928 06:32:40.836542 (+ 16112us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:40.836548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:40.836564 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:32:40.836607 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:40.836609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:40.836629 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:40.836730 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:40.845610 (+ 8880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.846632 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.848117 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.848895 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.848997 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.849058 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:40.849200 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:32:40.849244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16073,"thread_start_us":70,"threads_started":1}
I20260928 06:32:40.849324 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:40.817051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33761 (local address 127.0.0.1:59424)
0928 06:32:40.820695 (+ 3644us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:40.820749 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:40.820901 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:40.836742 (+ 15841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:40.836743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:40.836746 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:40.836795 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.836809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.846779 (+ 9970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.846781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.848007 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:40.848011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:40.849001 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:40.849002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:40.849093 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:40.849099 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:32:40.849177 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:32:40.849237 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3606,"thread_start_us":126,"threads_started":1}
I20260928 06:32:40.850111 3512 tablet_service.cc:1467] Processing CreateTablet for tablet 892276ec642f4478ba5630aa08822c5f (DEFAULT_TABLE table=test-table [id=33e54e160dde44e28a6067da747caa06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:32:40.850482 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 892276ec642f4478ba5630aa08822c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:32:40.854250 3892 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Bootstrap starting.
I20260928 06:32:40.855108 3892 tablet_bootstrap.cc:654] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Neither blocks nor log segments found. Creating new log.
I20260928 06:32:40.855716 3892 log.cc:824] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:32:40.856463 3892 tablet_bootstrap.cc:492] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: No bootstrap required, opened a new log
I20260928 06:32:40.856530 3892 ts_tablet_manager.cc:1397] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:32:40.859103 3892 raft_consensus.cc:348] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.859290 3892 raft_consensus.cc:374] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:32:40.859323 3892 raft_consensus.cc:729] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f31611e8f6341d684c6f26fbb14962b, State: Initialized, Role: FOLLOWER
I20260928 06:32:40.859468 3892 consensus_queue.cc:260] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:40.859742 3892 ts_tablet_manager.cc:1428] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:32:40.859786 3594 heartbeater.cc:503] Master 127.24.11.190:46683 was elected leader, sending a full tablet report...
Sep 28 06:32:40 dist-test-slave-38pg krb5kdc[2907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577160, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
W20260928 06:32:40.969099 3595 tablet.cc:2369] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:32:40.978173 3734 tablet.cc:2369] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:41.009687 3891 raft_consensus.cc:482] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:32:41.009928 3891 raft_consensus.cc:504] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:41.010566 3891 leader_election.cc:288] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdf23fb0b4b447d29579a02ea0a499a7 (127.24.11.130:33917), 8f31611e8f6341d684c6f26fbb14962b (127.24.11.129:33761)
I20260928 06:32:41.014545 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.010876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33761 (local address 127.24.11.131:54183)
0928 06:32:41.010949 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.010961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.011041 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.011277 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.011279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.011281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.011356 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.011358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.012270 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.012271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.013440 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.013442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.014302 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.014303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.014382 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.014385 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.014433 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.014471 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 06:32:41.014988 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.011334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52529 (local address 127.24.11.130:33917)
0928 06:32:41.011377 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.011379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.011652 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:32:41.011786 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.011787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.011796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.011841 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.011998 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.012604 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.014029 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.014550 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.014593 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.014603 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.014881 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.014922 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:32:41.015072 3885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.010907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54183 (local address 127.24.11.129:33761)
0928 06:32:41.010954 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.010956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.011012 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:32:41.011124 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.011125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.011134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.011206 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:32:41.011432 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.012183 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.013575 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.014158 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.014222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.014232 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.014950 (+ 718us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.014995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:32:41.015239 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.011409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33917 (local address 127.24.11.131:52529)
0928 06:32:41.011574 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.011585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:32:41.011704 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:32:41.011852 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:32:41.011852 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:32:41.011854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:32:41.011937 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.011938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.012699 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.012699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.013839 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:32:41.013841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.014693 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.014694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:32:41.014802 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.014806 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:32:41.015091 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 06:32:41.015135 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":120,"threads_started":1}
I20260928 06:32:41.015352 3548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892276ec642f4478ba5630aa08822c5f" candidate_uuid: "2f17883342b54a679950e917b643d094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f31611e8f6341d684c6f26fbb14962b" is_pre_election: true
I20260928 06:32:41.015539 3548 raft_consensus.cc:2454] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f17883342b54a679950e917b643d094 in term 0.
I20260928 06:32:41.015516 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892276ec642f4478ba5630aa08822c5f" candidate_uuid: "2f17883342b54a679950e917b643d094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" is_pre_election: true
I20260928 06:32:41.015643 3685 raft_consensus.cc:2454] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f17883342b54a679950e917b643d094 in term 0.
I20260928 06:32:41.015765 3756 leader_election.cc:302] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f17883342b54a679950e917b643d094, 8f31611e8f6341d684c6f26fbb14962b; no voters:
I20260928 06:32:41.015879 3891 raft_consensus.cc:2781] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:32:41.015921 3891 raft_consensus.cc:482] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:32:41.015942 3891 raft_consensus.cc:3037] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.016669 3891 raft_consensus.cc:504] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:41.016815 3891 leader_election.cc:288] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [CANDIDATE]: Term 1 election: Requested vote from peers fdf23fb0b4b447d29579a02ea0a499a7 (127.24.11.130:33917), 8f31611e8f6341d684c6f26fbb14962b (127.24.11.129:33761)
I20260928 06:32:41.016947 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892276ec642f4478ba5630aa08822c5f" candidate_uuid: "2f17883342b54a679950e917b643d094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf23fb0b4b447d29579a02ea0a499a7"
I20260928 06:32:41.016984 3548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "892276ec642f4478ba5630aa08822c5f" candidate_uuid: "2f17883342b54a679950e917b643d094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f31611e8f6341d684c6f26fbb14962b"
I20260928 06:32:41.017041 3685 raft_consensus.cc:3037] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.017059 3548 raft_consensus.cc:3037] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:32:41.017975 3685 raft_consensus.cc:2454] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f17883342b54a679950e917b643d094 in term 1.
I20260928 06:32:41.017977 3548 raft_consensus.cc:2454] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f17883342b54a679950e917b643d094 in term 1.
I20260928 06:32:41.018163 3756 leader_election.cc:302] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f17883342b54a679950e917b643d094, 8f31611e8f6341d684c6f26fbb14962b; no voters:
I20260928 06:32:41.018271 3891 raft_consensus.cc:2781] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:32:41.018453 3891 raft_consensus.cc:686] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 1 LEADER]: Becoming Leader. State: Replica: 2f17883342b54a679950e917b643d094, State: Running, Role: LEADER
I20260928 06:32:41.018553 3891 consensus_queue.cc:237] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } }
I20260928 06:32:41.019404 3384 catalog_manager.cc:5495] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f17883342b54a679950e917b643d094 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "2f17883342b54a679950e917b643d094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f17883342b54a679950e917b643d094" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45053 } health_report { overall_health: HEALTHY } } }
W20260928 06:32:41.020784 3869 tablet.cc:2369] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:32:41.051172 3884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.048621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.24.11.131:45053)
0928 06:32:41.048672 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:32:41.048675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:32:41.048686 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:32:41.048728 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:32:41.048729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:32:41.048750 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:32:41.048837 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:32:41.048986 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.049463 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.050362 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:32:41.050568 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:32:41.050649 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:32:41.050827 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:32:41.050934 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:32:41.051018 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:32:41.051058 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:32:41.051078 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:32:41.052897 3548 raft_consensus.cc:1261] T 892276ec642f4478ba5630aa08822c5f P 8f31611e8f6341d684c6f26fbb14962b [term 1 FOLLOWER]: Refusing update from remote peer 2f17883342b54a679950e917b643d094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:41.052928 3685 raft_consensus.cc:1261] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 1 FOLLOWER]: Refusing update from remote peer 2f17883342b54a679950e917b643d094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:32:41.053277 3891 consensus_queue.cc:1035] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f31611e8f6341d684c6f26fbb14962b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:41.053457 3896 consensus_queue.cc:1035] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf23fb0b4b447d29579a02ea0a499a7" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:32:41.055609 3899 mvcc.cc:205] Tried to move back new op lower bound from 7334204051670130688 to 7334204051532607488. Current Snapshot: MvccSnapshot[applied={T|T < 7334204051670130688}]
I20260928 06:32:41.056367 3901 mvcc.cc:205] Tried to move back new op lower bound from 7334204051670130688 to 7334204051532607488. Current Snapshot: MvccSnapshot[applied={T|T < 7334204051670130688}]
I20260928 06:32:41.056869 3900 mvcc.cc:205] Tried to move back new op lower bound from 7334204051670130688 to 7334204051532607488. Current Snapshot: MvccSnapshot[applied={T|T < 7334204051670130688}]
I20260928 06:32:41.059705 3383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:41.059826 3383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:32:41.060704 3383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53193d6fb8384e44aee2f3a92b8aba1a: Sending DeleteTablet for 3 replicas of tablet 892276ec642f4478ba5630aa08822c5f
I20260928 06:32:41.061072 3667 tablet_service.cc:1514] Processing DeleteTablet for tablet 892276ec642f4478ba5630aa08822c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:45046
I20260928 06:32:41.061183 3512 tablet_service.cc:1514] Processing DeleteTablet for tablet 892276ec642f4478ba5630aa08822c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:59424
I20260928 06:32:41.061496 3908 tablet_replica.cc:330] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: stopping tablet replica
I20260928 06:32:41.061509 3802 tablet_service.cc:1514] Processing DeleteTablet for tablet 892276ec642f4478ba5630aa08822c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:32:41 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:60242
I20260928 06:32:41.061709 3908 raft_consensus.cc:2229] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:32:41.061893 3908 raft_consensus.cc:2258] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:41.061939 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 3465
I20260928 06:32:41.063278 3908 ts_tablet_manager.cc:1905] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:32:41.065395 3908 ts_tablet_manager.cc:1918] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:32:41.065601 3908 log.cc:1196] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/892276ec642f4478ba5630aa08822c5f
I20260928 06:32:41.065982 3908 ts_tablet_manager.cc:1939] T 892276ec642f4478ba5630aa08822c5f P fdf23fb0b4b447d29579a02ea0a499a7: Deleting consensus metadata
I20260928 06:32:41.066574 3371 catalog_manager.cc:4841] TS fdf23fb0b4b447d29579a02ea0a499a7 (127.24.11.130:33917): tablet 892276ec642f4478ba5630aa08822c5f (table test-table [id=33e54e160dde44e28a6067da747caa06]) successfully deleted
I20260928 06:32:41.069700 3910 tablet_replica.cc:330] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: stopping tablet replica
I20260928 06:32:41.072171 3910 raft_consensus.cc:2229] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:32:41.072458 3910 raft_consensus.cc:2258] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:32:41.073087 3910 ts_tablet_manager.cc:1905] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:32:41.074486 3369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:33761 (error 108)
I20260928 06:32:41.075495 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.075346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:33761 (local address 127.0.0.1:59434)
0928 06:32:41.075398 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.075453 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:33761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:32:41.075521 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 3602
I20260928 06:32:41.077320 3910 ts_tablet_manager.cc:1918] T 892276ec642f4478ba5630aa08822c5f P 2f17883342b54a679950e917b643d094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 06:32:41.079506 3369 catalog_manager.cc:4584] TS 8f31611e8f6341d684c6f26fbb14962b (127.24.11.129:33761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 892276ec642f4478ba5630aa08822c5f: Network error: Client connection negotiation failed: client connection to 127.24.11.129:33761: connect: Connection refused (error 111)
I20260928 06:32:41.084091 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 3739
I20260928 06:32:41.089789 3882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:32:41.089625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45053 (local address 127.0.0.1:60256)
0928 06:32:41.089689 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:32:41.089723 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.131:45053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:32:41.089984 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 3352
I20260928 06:32:41.096722 24622 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:32:41.431736 24622 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:32:41.431862 24622 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:32:41.777093 24622 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:32:41.777 UTC [2937] LOG: received smart shutdown request
2026-09-28 06:32:41.781 UTC [2937] LOG: background worker "logical replication launcher" (PID 2946) exited with exit code 1
2026-09-28 06:32:41.781 UTC [2941] LOG: shutting down
2026-09-28 06:32:41.968 UTC [2937] LOG: database system is shut down
2026-09-28T06:32:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50112 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:32:42 dist-test-slave-38pg krb5kdc[3920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:32:42 dist-test-slave-38pg krb5kdc[3920](info): set up 2 sockets
Sep 28 06:32:42 dist-test-slave-38pg krb5kdc[3920](info): commencing operation
krb5kdc: starting...
W20260928 06:32:43.138548 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:32:43 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:32:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangeradmin_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangeradmin_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangerlookup_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangerlookup_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
I20260928 06:32:43.304468 24622 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwumdS4/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/postgres -l logfile start
2026-09-28 06:32:45.878 UTC [3950] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:32:45.878 UTC [3950] LOG: listening on IPv4 address "127.24.11.148", port 50811
2026-09-28 06:32:45.882 UTC [3950] LOG: listening on Unix socket "/tmp/.s.PGSQL.50811"
2026-09-28 06:32:45.903 UTC [3953] LOG: database system was shut down at 2026-09-28 06:32:44 UTC
2026-09-28 06:32:45.907 UTC [3950] LOG: database system is ready to accept connections
I20260928 06:32:46.854614 24622 mini_postgres.cc:96] Postgres bound to 50811
2026-09-28 06:32:46.860 UTC [3961] FATAL: database "slave" does not exist
127.24.11.148:50811 - accepting connections
I20260928 06:32:46.861176 24622 mini_ranger.cc:161] Starting Ranger...
I20260928 06:32:46.875944 24622 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:32:47.370000 24622 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:32:47.370075 24622 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin
2026-09-28 06:32:47,645 [I] DB FLAVOR :POSTGRES
2026-09-28 06:32:47,645 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:32:47,645 [I] Checking connection..
2026-09-28 06:32:47,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:32:47,917 [I] Checking connection passed.
2026-09-28 06:32:47,917 [I] --------- Verifying version history table ---------
2026-09-28 06:32:47,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:32:48,176 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:32:48,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:32:48,449 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:32:48,450 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:32:48,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:32:48.680 UTC [4025] WARNING: there is no transaction in progress
2026-09-28 06:32:48,687 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:32:48,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:32:48,949 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:32:48,949 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:32:48,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:32:49,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:32:49,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-38pg.c.gcp-upstream.internal','N') ;"
2026-09-28 06:32:49,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:32:49,967 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:32:49,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:32:50,232 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:32:50,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:32:50,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:32:50,732 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:32:50,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:32:51.754 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.773 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.791 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.808 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.825 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.846 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.914 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.921 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.937 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.947 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.958 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.973 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.984 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.991 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:51.999 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:52.006 UTC [4133] WARNING: there is no transaction in progress
2026-09-28 06:32:52,678 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:32:52,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:32:52,943 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:32:52,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:32:53,202 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:32:53,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:32:53,454 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:32:53,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:32:53,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:32:53,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-38pg.c.gcp-upstream.internal';"
2026-09-28 06:32:54,214 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:32:54,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:32:54,488 [I] DB_PATCHES have already been applied
I20260928 06:32:54.494673 24622 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwumdS4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:32:54.494756 24622 mini_ranger.cc:193] Using host: 127.24.11.148
I20260928 06:32:54.497826 24622 mini_ranger.cc:227] Ranger admin URL: http://127.24.11.148:42615
Sep 28, 2026 6:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:32:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:32:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:32:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:32:55.597190 24622 mini_ranger.cc:160] Time spent starting Ranger: real 8.736s user 0.000s sys 0.009s
Sep 28, 2026 6:33:16 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:33:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:33:16 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:33:16 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:33:17 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:33:18.162448 24622 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.11.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.11.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangerkms_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.11.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/rangerkms_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.11.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.11.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.11.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/HTTP_127.24.11.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:33:18 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:33:18.247354 24622 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:33:18.257266 24622 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:33:19.145860 24622 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:33:19.145960 24622 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms
I20260928 06:33:19.145982 24622 mini_ranger_kms.cc:227] Using postgres at 127.24.11.148:50811
2026-09-28 06:33:19,324 [I] DB FLAVOR :POSTGRES
2026-09-28 06:33:19,325 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:33:19,325 [I] Checking connection
2026-09-28 06:33:19,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:33:19,609 [I] connection success
2026-09-28 06:33:19,610 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:33:19,610 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:33:19,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:33:19,860 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:33:19,861 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:33:19,861 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:33:19,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.11.148:50811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:33:20,150 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:33:20.646322 24622 mini_ranger_kms.cc:310] Created kms service
I20260928 06:33:20.844417 24622 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:33:20.892500 24622 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:33:20 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577198, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:21.308564 24622 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:33:21.308671 24622 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwumdS4/build/release/bin/postgresql.jar:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwumdS4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:33:21.308691 24622 mini_ranger_kms.cc:240] Using host: 127.24.11.148
I20260928 06:33:21.311889 24622 mini_ranger_kms.cc:276] Ranger KMS PID: 4314
I20260928 06:33:21.311973 24622 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.11.148:51505
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:33:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 06:33:23 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:23.579288 24622 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.332s user 0.000s sys 0.157s
I20260928 06:33:23.579416 24622 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:33:23.579485 24622 mini_ranger_kms.cc:401] 127.24.11.148:51505/kms/v1/keys
Sep 28 06:33:23 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
Sep 28 06:33:24 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577203, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.11.148@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
Sep 28 06:33:26 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:33:26.596320 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38131
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:38131
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:26.677698 4364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:26.677924 4364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:26.677946 4364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:26.679143 4364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:26.679188 4364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:26.679203 4364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:26.679247 4364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:26.679318 4364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:26.681051 4364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.11.148:51505/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:38131
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:26.681298 4364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:26.681600 4364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:26.686837 4364 server_base.cc:1034] running on GCE node
W20260928 06:33:26.686820 4370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:26.687024 4372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:26.686812 4369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:26.687336 4364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:26.687682 4364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:26.688853 4364 hybrid_clock.cc:648] HybridClock initialized: now 1790577206688837 us; error 32 us; skew 500 ppm
Sep 28 06:33:26 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:26.696206 4364 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:26.696750 4364 webserver.cc:460] Webserver started at http://127.24.11.190:38677/ using document root <none> and password file <none>
I20260928 06:33:26.697026 4364 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:26.697088 4364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:26.697242 4364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:33:26 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577206, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:26.759989 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "efe0193525444c7680b8ba0eae0d13b2"
format_stamp: "Formatted at 2026-09-28 06:33:26 on dist-test-slave-38pg"
server_key: "871e306376c0d2ae1e65819d9d767d8c"
server_key_iv: "1c8aea4067283bcd4fa8d0deb395fece"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:26.760399 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "efe0193525444c7680b8ba0eae0d13b2"
format_stamp: "Formatted at 2026-09-28 06:33:26 on dist-test-slave-38pg"
server_key: "871e306376c0d2ae1e65819d9d767d8c"
server_key_iv: "1c8aea4067283bcd4fa8d0deb395fece"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:26.761904 4364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 06:33:26.773952 4379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:26.774192 4364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:26.774297 4364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "efe0193525444c7680b8ba0eae0d13b2"
format_stamp: "Formatted at 2026-09-28 06:33:26 on dist-test-slave-38pg"
server_key: "871e306376c0d2ae1e65819d9d767d8c"
server_key_iv: "1c8aea4067283bcd4fa8d0deb395fece"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:26.774422 4364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:26.795575 4364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:26.797696 4364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:26.797889 4364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:26.802983 4364 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:38131
I20260928 06:33:26.803064 4431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:38131 every 8 connection(s)
I20260928 06:33:26.803434 4364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:26.804221 4432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:26.807339 4432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Bootstrap starting.
I20260928 06:33:26.808050 4432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:26.808374 4432 log.cc:824] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:26.809151 4432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: No bootstrap required, opened a new log
I20260928 06:33:26.812490 4432 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:26.812712 4432 raft_consensus.cc:374] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:26.812754 4432 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe0193525444c7680b8ba0eae0d13b2, State: Initialized, Role: FOLLOWER
I20260928 06:33:26.812777 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4364
I20260928 06:33:26.812878 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:33:26.812973 4432 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:26.813082 24622 external_mini_cluster.cc:1428] Setting key ad341a495ceaf884344fabb7b75c57a6
I20260928 06:33:26.813112 4432 raft_consensus.cc:388] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:26.813155 4432 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:26.813249 4432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:26.814121 4432 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:26.814301 4432 leader_election.cc:302] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efe0193525444c7680b8ba0eae0d13b2; no voters:
I20260928 06:33:26.814455 4432 leader_election.cc:288] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:26.814509 4437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:26.814739 4437 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 LEADER]: Becoming Leader. State: Replica: efe0193525444c7680b8ba0eae0d13b2, State: Running, Role: LEADER
I20260928 06:33:26.814785 4432 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:26.814906 4437 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:26.815250 4437 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efe0193525444c7680b8ba0eae0d13b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } } }
I20260928 06:33:26.815436 4437 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:26.815294 4438 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe0193525444c7680b8ba0eae0d13b2. Latest consensus state: current_term: 1 leader_uuid: "efe0193525444c7680b8ba0eae0d13b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } } }
I20260928 06:33:26.816463 4438 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:26.816500 4432 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:33:26.818867 4452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:26.820035 4452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:33:26 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:26.828140 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:26.813615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.24.11.190:38131)
0928 06:33:26.813811 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:26.813821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:26.813856 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:33:26.814006 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:26.814012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:26.814058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:26.814210 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:26.815045 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.816705 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.818293 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:26.820012 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:26.820155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:26.823744 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:26.823749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:26.823749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:26.823761 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:26.826054 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:26.826563 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:26.826566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:26.826567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:26.826703 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:26.826923 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:26.826924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:26.826924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:26.827293 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:26.827424 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:26.827696 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:33:26.827849 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260928 06:33:26.830068 4452 catalog_manager.cc:1347] Generated new cluster ID: 0ef3536cd3664060b51e060322943c0c
I20260928 06:33:26.830138 4452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:26.855907 4452 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:26.856645 4452 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:26.860080 4452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Generated new TSK 0
I20260928 06:33:26.860363 4452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:33:27.588327 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:27.677380 4477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:27.677635 4477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:27.677662 4477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:27.678833 4477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:27.678872 4477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:27.678898 4477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:27.680327 4477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:27.680522 4477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:27.680694 4477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:27.686486 4483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:27.686507 4485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:27.686474 4482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:27.686797 4477 server_base.cc:1034] running on GCE node
I20260928 06:33:27.687057 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:27.687311 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:27.688475 4477 hybrid_clock.cc:648] HybridClock initialized: now 1790577207688465 us; error 27 us; skew 500 ppm
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:27.691121 4477 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:27.691610 4477 webserver.cc:460] Webserver started at http://127.24.11.129:44931/ using document root <none> and password file <none>
I20260928 06:33:27.691818 4477 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:27.691860 4477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:27.692009 4477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:27.710243 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "49ef8ba2e53d456ebe884241e3d41de9"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "059eadff34b9162a81f1d3785b40b053"
server_key_iv: "10965eb550b41c49ad7d3ea95b2029b3"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:27.710738 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "49ef8ba2e53d456ebe884241e3d41de9"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "059eadff34b9162a81f1d3785b40b053"
server_key_iv: "10965eb550b41c49ad7d3ea95b2029b3"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:27.712603 4477 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:33:27.729511 4492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:27.729789 4477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:27.729893 4477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "49ef8ba2e53d456ebe884241e3d41de9"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "059eadff34b9162a81f1d3785b40b053"
server_key_iv: "10965eb550b41c49ad7d3ea95b2029b3"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:27.729985 4477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:27.759546 4477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:27.763758 4477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:27.763944 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:27.764200 4477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:27.764528 4477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:27.764585 4477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:27.764627 4477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:27.764653 4477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:27.771924 4477 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39459
I20260928 06:33:27.771981 4607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39459 every 8 connection(s)
I20260928 06:33:27.772419 4477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:27.773890 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4477
I20260928 06:33:27.773985 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:27.774186 24622 external_mini_cluster.cc:1428] Setting key 2fb487d51e933c00abdbf952716a9a79
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:27.785000 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:27.773342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:40523 (local address 127.24.11.190:38131)
0928 06:33:27.773474 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:27.773476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:27.774446 (+ 970us) server_negotiation.cc:407] Connection header received
0928 06:33:27.776822 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:27.776824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:27.776841 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:27.776906 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:27.777803 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.778320 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.779475 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.779686 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.779773 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:27.782003 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:27.782005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:27.782006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:27.782014 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:27.783550 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:27.783978 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:27.783980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:27.783980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:27.784048 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:27.784304 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:27.784305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:27.784305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:27.784434 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:27.784513 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:27.784789 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:33:27.784894 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 06:33:27.785311 4611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:27.773327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.129:40523)
0928 06:33:27.773563 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:27.773606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:27.776633 (+ 3027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:27.777044 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:27.777049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:27.777329 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:27.777696 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:27.777702 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.778419 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.778420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:27.779364 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:27.779366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:27.779783 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:27.779784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:27.779963 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:27.780544 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:27.780553 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:27.781899 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:27.783655 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:27.783662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:27.783665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:27.783892 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:27.784115 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:27.784116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:27.784117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:27.784230 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:27.784501 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:27.784508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:27.784647 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 06:33:27.784754 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:33:27.784909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260928 06:33:27.785821 4608 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:27.785928 4608 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:27.786113 4608 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:27.786670 4396 ts_manager.cc:194] Registered new tserver with Master: 49ef8ba2e53d456ebe884241e3d41de9 (127.24.11.129:39459)
I20260928 06:33:27.787360 4396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:40523
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:33:27.812036 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:27.934645 4614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:27.935437 4614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:27.935570 4614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:27.938634 4614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:27.938773 4614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:27.938879 4614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:27.942348 4614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:27.942934 4614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:27.943300 4614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:27.950438 4624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:27.950448 4622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:27.950680 4621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:27.963078 4614 server_base.cc:1034] running on GCE node
I20260928 06:33:27.963382 4614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:27.963717 4614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:27.964880 4614 hybrid_clock.cc:648] HybridClock initialized: now 1790577207964855 us; error 37 us; skew 500 ppm
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:27.968163 4614 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:27.968751 4614 webserver.cc:460] Webserver started at http://127.24.11.130:33397/ using document root <none> and password file <none>
I20260928 06:33:27.969046 4614 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:27.969111 4614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:27.969244 4614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:33:27 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:27.992242 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "918776aece0941d1826a0baaf27f05ba"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "7e1b2fe644f114f4ec5b3c18f74b96ca"
server_key_iv: "f4892fd58c63471483abe57f1e9d9ee4"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:27.992821 4614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "918776aece0941d1826a0baaf27f05ba"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "7e1b2fe644f114f4ec5b3c18f74b96ca"
server_key_iv: "f4892fd58c63471483abe57f1e9d9ee4"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:27.998039 4614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260928 06:33:28.029704 4631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.030023 4614 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260928 06:33:28.030144 4614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "918776aece0941d1826a0baaf27f05ba"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "7e1b2fe644f114f4ec5b3c18f74b96ca"
server_key_iv: "f4892fd58c63471483abe57f1e9d9ee4"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.030260 4614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:28.041435 4614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:28.044116 4614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:28.044278 4614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:28.044561 4614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:28.044929 4614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:28.044973 4614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.045004 4614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:28.045022 4614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.052556 4614 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33203
I20260928 06:33:28.052655 4745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33203 every 8 connection(s)
I20260928 06:33:28.052956 4614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:28.054925 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4614
I20260928 06:33:28.055032 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:28.055217 24622 external_mini_cluster.cc:1428] Setting key 543105cc6edb3edec6711632dd61bce0
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577207, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:28.064774 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.053795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:48029 (local address 127.24.11.190:38131)
0928 06:33:28.053840 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:28.053842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:28.054109 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:33:28.054651 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:28.054653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:28.054670 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:28.054743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:28.055501 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.056390 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.058201 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.058424 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.058512 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.061124 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:28.061126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:28.061127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:28.061136 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:28.063032 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.063535 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.063536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.063536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.063598 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.063934 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.063935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.063935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.064078 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:28.064121 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:28.064497 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 06:33:28.064590 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
I20260928 06:33:28.065548 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.053809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.130:48029)
0928 06:33:28.053984 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:28.054020 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:28.054542 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:28.054883 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:28.054886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:28.055143 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:28.055344 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.055348 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.056515 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.056516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.058092 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.058095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.058534 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.058535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.058665 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.059270 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:28.059281 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:28.061016 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:28.063129 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.063136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.063139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.063439 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.063671 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.063671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.063672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.063799 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.064171 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:28.064178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:28.064344 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 06:33:28.064958 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 06:33:28.065285 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260928 06:33:28.066006 4746 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:28.066102 4746 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.066296 4746 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:28.066648 4396 ts_manager.cc:194] Registered new tserver with Master: 918776aece0941d1826a0baaf27f05ba (127.24.11.130:33203)
I20260928 06:33:28.067154 4396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:48029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:33:28.098284 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:28.196004 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:28.196254 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:28.196295 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:28.197477 4751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:28.197566 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:28.197615 4751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:28.199297 4751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:28.199563 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:28.199785 4751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:28.204699 4756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.204699 4757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.205310 4759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.205631 4751 server_base.cc:1034] running on GCE node
I20260928 06:33:28.205835 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:28.206158 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:28.207314 4751 hybrid_clock.cc:648] HybridClock initialized: now 1790577208207303 us; error 26 us; skew 500 ppm
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:28.209923 4751 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:28.210393 4751 webserver.cc:460] Webserver started at http://127.24.11.131:40829/ using document root <none> and password file <none>
I20260928 06:33:28.210632 4751 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:28.210702 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:28.210843 4751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:28.243449 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6"
format_stamp: "Formatted at 2026-09-28 06:33:28 on dist-test-slave-38pg"
server_key: "d81cddbee8978b5b124e8efcf7717a25"
server_key_iv: "755d25689b223d370d3b9553722d6a05"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.244292 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6"
format_stamp: "Formatted at 2026-09-28 06:33:28 on dist-test-slave-38pg"
server_key: "d81cddbee8978b5b124e8efcf7717a25"
server_key_iv: "755d25689b223d370d3b9553722d6a05"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.246109 4751 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:33:28.264901 4766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.265378 4751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 06:33:28.265501 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6"
format_stamp: "Formatted at 2026-09-28 06:33:28 on dist-test-slave-38pg"
server_key: "d81cddbee8978b5b124e8efcf7717a25"
server_key_iv: "755d25689b223d370d3b9553722d6a05"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.265632 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:28.283777 4751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:28.286682 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:28.286887 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:28.287177 4751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:28.287611 4751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:28.287668 4751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.287708 4751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:28.287738 4751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.303308 4751 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37001
I20260928 06:33:28.303846 4879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37001 every 8 connection(s)
I20260928 06:33:28.303891 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:28.308665 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4751
I20260928 06:33:28.308789 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:28.308986 24622 external_mini_cluster.cc:1428] Setting key f236f794c2bda1713864a4d6dd5b500f
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:28.316118 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.305240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:47019 (local address 127.24.11.190:38131)
0928 06:33:28.305284 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:28.305287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:28.305484 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:33:28.306202 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:28.306204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:28.306216 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:28.306246 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:28.307089 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.307893 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.309391 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.309707 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.309755 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.312448 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:28.312450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:28.312450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:28.312458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:28.314281 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.314837 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.314838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.314839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.314912 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.315239 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.315240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.315241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.315397 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:28.315489 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:28.315908 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 06:33:28.316018 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:28.318133 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.304970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.131:47019)
0928 06:33:28.305359 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:28.305404 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:28.306074 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:28.306434 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:28.306439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:28.306692 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:28.306978 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.306984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.308012 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.308013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.309281 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.309283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.309876 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.309877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.310022 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.310644 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:28.310654 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:28.312323 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:28.314416 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.314424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.314426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.314751 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.315003 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.315005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.315005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.315164 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.315483 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:28.315489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:28.315731 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 06:33:28.315869 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 06:33:28.317876 (+ 2007us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":77,"threads_started":1}
I20260928 06:33:28.318799 4880 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:28.318919 4880 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.319115 4880 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:28.319567 4396 ts_manager.cc:194] Registered new tserver with Master: 2a3c2b35a2f64cf189c86e51c1cb93f6 (127.24.11.131:37001)
I20260928 06:33:28.320137 4396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:47019
I20260928 06:33:28.320712 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:28.321412 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4477
I20260928 06:33:28.328797 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4614
I20260928 06:33:28.335547 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4751
I20260928 06:33:28.340118 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4364
I20260928 06:33:28.347466 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38131
--webserver_interface=127.24.11.190
--webserver_port=38677
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:38131
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:28.429469 4883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:28.429709 4883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:28.429731 4883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:28.430850 4883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:28.430886 4883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:28.430900 4883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:28.430912 4883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:28.430925 4883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:28.432499 4883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:38131
--ranger_config_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:28.432679 4883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:28.432840 4883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:28.437589 4889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.437654 4883 server_base.cc:1034] running on GCE node
W20260928 06:33:28.437516 4888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.437525 4891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.437970 4883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:28.438273 4883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:28.439441 4883 hybrid_clock.cc:648] HybridClock initialized: now 1790577208439444 us; error 41 us; skew 500 ppm
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:28.441963 4883 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:28.442426 4883 webserver.cc:460] Webserver started at http://127.24.11.190:38677/ using document root <none> and password file <none>
I20260928 06:33:28.442638 4883 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:28.442710 4883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:28.496912 4883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.497711 4899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.497942 4883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:28.498046 4883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "efe0193525444c7680b8ba0eae0d13b2"
format_stamp: "Formatted at 2026-09-28 06:33:26 on dist-test-slave-38pg"
server_key: "871e306376c0d2ae1e65819d9d767d8c"
server_key_iv: "1c8aea4067283bcd4fa8d0deb395fece"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.498437 4883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:28.508399 4883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:28.510337 4883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:28.510509 4883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:28.515025 4883 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:38131
I20260928 06:33:28.515064 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:38131 every 8 connection(s)
I20260928 06:33:28.515455 4883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:28.516649 4952 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:28.517303 4952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Bootstrap starting.
I20260928 06:33:28.523322 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4883
I20260928 06:33:28.523597 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:39459
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:38131
--webserver_port=44931
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:28.525740 4952 log.cc:824] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:28.527411 4952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:28.527709 4952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Bootstrap complete.
I20260928 06:33:28.530228 4952 raft_consensus.cc:348] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:28.530465 4952 raft_consensus.cc:729] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe0193525444c7680b8ba0eae0d13b2, State: Initialized, Role: FOLLOWER
I20260928 06:33:28.530634 4952 consensus_queue.cc:260] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:28.530718 4952 raft_consensus.cc:388] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:28.530786 4952 raft_consensus.cc:482] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:28.530843 4952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:33:28.531872 4952 raft_consensus.cc:504] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:28.531961 4952 leader_election.cc:302] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efe0193525444c7680b8ba0eae0d13b2; no voters:
I20260928 06:33:28.532172 4952 leader_election.cc:288] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:33:28.532207 4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:33:28.532330 4956 raft_consensus.cc:686] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [term 2 LEADER]: Becoming Leader. State: Replica: efe0193525444c7680b8ba0eae0d13b2, State: Running, Role: LEADER
I20260928 06:33:28.532416 4956 consensus_queue.cc:237] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } }
I20260928 06:33:28.532444 4952 sys_catalog.cc:564] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:28.532701 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "efe0193525444c7680b8ba0eae0d13b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } } }
I20260928 06:33:28.532822 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:28.532693 4957 sys_catalog.cc:455] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efe0193525444c7680b8ba0eae0d13b2. Latest consensus state: current_term: 2 leader_uuid: "efe0193525444c7680b8ba0eae0d13b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efe0193525444c7680b8ba0eae0d13b2" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38131 } } }
I20260928 06:33:28.532990 4957 sys_catalog.cc:458] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:28.533424 4961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:28.533952 4961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:28.534096 4961 catalog_manager.cc:1259] Loaded cluster ID: 0ef3536cd3664060b51e060322943c0c
I20260928 06:33:28.534138 4961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:28.534273 4952 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:33:28.538022 4961 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:28.538558 4961 catalog_manager.cc:5879] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Loaded TSK: 0
I20260928 06:33:28.538856 4961 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:33:28.637969 4954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:28.638255 4954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:28.638305 4954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:28.640044 4954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:28.640174 4954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:28.640262 4954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:28.642623 4954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:39459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=44931
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:28.643179 4954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:28.643488 4954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:28.649839 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.649874 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.649999 4954 server_base.cc:1034] running on GCE node
W20260928 06:33:28.650014 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.650569 4954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:28.650925 4954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:28.652127 4954 hybrid_clock.cc:648] HybridClock initialized: now 1790577208652124 us; error 36 us; skew 500 ppm
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:28.654826 4954 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:28.655385 4954 webserver.cc:460] Webserver started at http://127.24.11.129:44931/ using document root <none> and password file <none>
I20260928 06:33:28.655648 4954 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:28.655732 4954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:28.673985 4954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.678007 5000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.678257 4954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260928 06:33:28.678365 4954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "49ef8ba2e53d456ebe884241e3d41de9"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "059eadff34b9162a81f1d3785b40b053"
server_key_iv: "10965eb550b41c49ad7d3ea95b2029b3"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.678838 4954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:28.709154 4954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:28.711859 4954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:28.712087 4954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:28.712397 4954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:28.712800 4954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:28.712889 4954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.712951 4954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:28.712996 4954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.719473 4954 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39459
I20260928 06:33:28.719990 4954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:28.722982 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 4954
I20260928 06:33:28.723254 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:33203
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:38131
--webserver_port=33397
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:28.730691 5113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39459 every 8 connection(s)
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:28.748833 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.731775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:60683 (local address 127.24.11.190:38131)
0928 06:33:28.737753 (+ 5978us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:28.737764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:28.737803 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:33:28.737966 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:28.737972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:28.738028 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:28.738186 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:28.738802 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.739712 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.740755 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.741220 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.741323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.744181 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:28.744186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:28.744187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:28.744201 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:28.746413 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.746976 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.746980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.746981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.747105 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.747409 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.747410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.747410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.747738 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:28.747839 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:28.748478 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 06:33:28.748601 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5904,"thread_start_us":80,"threads_started":1}
I20260928 06:33:28.749419 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.731420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.129:60683)
0928 06:33:28.736768 (+ 5348us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:28.736848 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:28.737488 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:28.738231 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:28.738235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:28.738462 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:28.738662 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.738668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.739840 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.739841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.740649 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.740652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.741417 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.741418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.741512 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.742218 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:28.742228 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:28.744031 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:28.746523 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.746530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.746533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.746872 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.747188 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.747188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.747189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.747328 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.747870 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:28.747877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:28.748037 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:33:28.748153 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 06:33:28.748324 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5292,"thread_start_us":82,"threads_started":1}
I20260928 06:33:28.750574 5114 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:28.750674 5114 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.750861 5114 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:28.751753 5114 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:38131 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:33:28.751818 5114 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.751878 5114 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:28.752236 5114 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.752315 5114 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:28.752617 5114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:33:28.838876 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:28.839166 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:28.839210 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:28.841087 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:28.841150 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:28.841193 5116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:28.843565 5116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:33203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=33397
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:28.843930 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:28.844247 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:28.850317 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.850281 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:28.850482 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:28.850742 5116 server_base.cc:1034] running on GCE node
I20260928 06:33:28.850994 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:28.851322 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:28.852515 5116 hybrid_clock.cc:648] HybridClock initialized: now 1790577208852502 us; error 31 us; skew 500 ppm
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:28.855872 5116 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:28.856478 5116 webserver.cc:460] Webserver started at http://127.24.11.130:33397/ using document root <none> and password file <none>
I20260928 06:33:28.856747 5116 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:28.856848 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:28.872715 5116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:28.877332 5134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.877661 5116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 06:33:28.877777 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "918776aece0941d1826a0baaf27f05ba"
format_stamp: "Formatted at 2026-09-28 06:33:27 on dist-test-slave-38pg"
server_key: "7e1b2fe644f114f4ec5b3c18f74b96ca"
server_key_iv: "f4892fd58c63471483abe57f1e9d9ee4"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:28.878288 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:28.889810 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:28.892391 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:28.892580 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:28.892905 5116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:28.893309 5116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:28.893365 5116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.893399 5116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:28.893428 5116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:28.901204 5116 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33203
I20260928 06:33:28.901736 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:28.902328 5247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33203 every 8 connection(s)
I20260928 06:33:28.906669 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5116
I20260928 06:33:28.906961 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.11.148:51505/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:37001
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:38131
--webserver_port=40829
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:40365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:33:28 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577208, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:28.916134 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.902910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:43669 (local address 127.24.11.190:38131)
0928 06:33:28.902965 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:28.902967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:28.902977 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:28.903394 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:28.903395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:28.903407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:28.903464 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:28.904153 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.904844 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.906134 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.906356 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.906443 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.912644 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:28.912646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:28.912646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:28.912655 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:28.914237 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.914868 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.914870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.914870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.914949 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:28.915312 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:28.915313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:28.915314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:28.915474 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:28.915557 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:28.915938 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 06:33:28.916035 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:33:28.916379 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:28.902519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.130:43669)
0928 06:33:28.902690 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:28.902734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:28.903281 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:28.903541 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:28.903545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:28.903739 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:28.904044 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.904052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.904962 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.904964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.906028 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:28.906031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:28.908600 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:28.908603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:28.908763 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:28.909420 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:28.909430 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:28.912511 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:28.914372 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.914379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.914382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.914756 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.915048 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:28.915049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:28.915049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:28.915220 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:28.915558 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:28.915566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:28.915763 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 06:33:28.915895 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:33:28.916084 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 06:33:28.917004 5248 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:28.917120 5248 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.917320 5248 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:28.918030 5248 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:38131 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:33:28.918115 5248 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.918184 5248 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:28.918529 5248 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:28.918601 5248 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:28.918956 5248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:33:29.016551 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:29.016799 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:29.016858 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:29.018167 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:29.018236 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:29.018283 5251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:29.019824 5251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:40365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.11.148:51505/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:37001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=40829
--tserver_master_addrs=127.24.11.190:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:29.020097 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:29.020391 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:29.025494 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:29.025470 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:29.025470 5256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:29.026105 5251 server_base.cc:1034] running on GCE node
I20260928 06:33:29.026325 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:29.026644 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:29.027863 5251 hybrid_clock.cc:648] HybridClock initialized: now 1790577209027841 us; error 27 us; skew 500 ppm
Sep 28 06:33:29 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577209, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:29.030925 5251 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:29.031479 5251 webserver.cc:460] Webserver started at http://127.24.11.131:40829/ using document root <none> and password file <none>
I20260928 06:33:29.031780 5251 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:29.031852 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:33:29 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577209, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:29.046831 5251 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:29.047838 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:29.048025 5251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:29.048120 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6"
format_stamp: "Formatted at 2026-09-28 06:33:28 on dist-test-slave-38pg"
server_key: "d81cddbee8978b5b124e8efcf7717a25"
server_key_iv: "755d25689b223d370d3b9553722d6a05"
server_key_version: "kuduclusterkey@0"
I20260928 06:33:29.048529 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:29.061107 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:29.063889 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:29.064062 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:29.064337 5251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:29.064677 5251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:29.064718 5251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:29.064751 5251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:29.064775 5251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:29.072718 5251 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37001
I20260928 06:33:29.072813 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37001 every 8 connection(s)
I20260928 06:33:29.073379 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:29 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577209, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:29.083582 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5251
I20260928 06:33:29.085703 5118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:29.075468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:34527 (local address 127.24.11.190:38131)
0928 06:33:29.075512 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:29.075514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:29.075792 (+ 278us) server_negotiation.cc:407] Connection header received
0928 06:33:29.076464 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:29.076467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:29.076481 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:29.076541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:29.077211 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.078002 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.079206 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.079439 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.079531 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:29.081771 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:29.081773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:29.081773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:29.081781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:29.083617 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:29.084433 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:29.084435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:29.084435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:29.084525 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:29.084859 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:29.084861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:29.084861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:29.085021 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:29.085067 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:29.085487 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 06:33:29.085608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:29.086498 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:29.075467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38131 (local address 127.24.11.131:34527)
0928 06:33:29.075623 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:29.075661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:29.076317 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:29.076618 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:29.076623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:29.076859 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:29.077101 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:29.077106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.078122 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.078123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:29.079105 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:29.079107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.079562 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.079564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:29.079688 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:29.080314 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:29.080323 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:29.081672 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:29.083749 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:29.083756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:29.083760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:29.084343 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:29.084607 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:29.084607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:29.084608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:29.084762 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:29.085118 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:29.085124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:29.085302 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 06:33:29.086067 (+ 765us) client_negotiation.cc:240] Negotiation successful
0928 06:33:29.086259 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 06:33:29.087296 5382 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38131
I20260928 06:33:29.087415 5382 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:29.087646 5382 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:29.088652 5382 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:38131 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:33:29.088734 5382 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:29.088866 5382 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:29.089248 5382 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:29.089334 5382 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
W20260928 06:33:29.089663 5382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:33:29.092207 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:29.084026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.24.11.190:38131)
0928 06:33:29.084185 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:29.084187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:29.084350 (+ 163us) server_negotiation.cc:407] Connection header received
0928 06:33:29.084419 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:29.084420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:29.084434 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:29.084525 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:29.085278 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.086063 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.087121 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:29.087438 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:29.087502 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:29.089415 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:29.089416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:29.089416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:29.089423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:29.090984 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:29.091340 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:29.091341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:29.091342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:29.091397 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:29.091613 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:29.091614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:29.091614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:29.091735 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:29.091963 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:29.092003 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:33:29.092099 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
Sep 28 06:33:29 dist-test-slave-38pg krb5kdc[3920](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577209, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:29.752794 5114 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:29.752961 5114 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:29.753434 4915 ts_manager.cc:194] Registered new tserver with Master: 49ef8ba2e53d456ebe884241e3d41de9 (127.24.11.129:39459)
I20260928 06:33:29.754493 4915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:60683
I20260928 06:33:29.919173 5248 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:29.919346 5248 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:29.919715 4915 ts_manager.cc:194] Registered new tserver with Master: 918776aece0941d1826a0baaf27f05ba (127.24.11.130:33203)
I20260928 06:33:29.920272 4915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:43669
Sep 28 06:33:30 dist-test-slave-38pg krb5kdc[3920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577209, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for HTTP/127.24.11.148@KRBTEST.COM
I20260928 06:33:30.090210 5382 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:30.090397 5382 heartbeater.cc:511] Master 127.24.11.190:38131 requested a full tablet report, sending...
I20260928 06:33:30.090793 4915 ts_manager.cc:194] Registered new tserver with Master: 2a3c2b35a2f64cf189c86e51c1cb93f6 (127.24.11.131:37001)
I20260928 06:33:30.091293 4915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:34527
I20260928 06:33:30.098498 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:30.108294 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.099619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39846 (local address 127.24.11.190:38131)
0928 06:33:30.099971 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.099973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.099981 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:30.100002 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.100003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.100014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.100896 (+ 882us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:30.100932 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.101835 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.102910 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.103217 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.103576 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.104802 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:30.104804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:30.104804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:30.104810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:30.106710 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:30.107145 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:30.107146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:30.107146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:30.107207 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:30.107403 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:30.107403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:30.107404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:30.107541 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:30.108061 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.108106 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.108187 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":61,"threads_started":1}
I20260928 06:33:30.110795 4915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:30.113655 4915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:30.126122 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.116999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:37001 (local address 127.0.0.1:56510)
0928 06:33:30.121662 (+ 4663us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.121713 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:30.121792 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:30.122158 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:30.122161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:30.122164 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:30.122241 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.122243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.123524 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.123525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.124767 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.124770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.125835 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.125836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.125911 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.125924 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:33:30.125990 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:33:30.126040 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4628,"thread_start_us":96,"threads_started":1}
I20260928 06:33:30.126696 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.117454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.24.11.131:37001)
0928 06:33:30.117592 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.117596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.121889 (+ 4293us) server_negotiation.cc:407] Connection header received
0928 06:33:30.121932 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.121935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.121953 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.122037 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:30.122326 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.123414 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.124867 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.125662 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.125743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.125801 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.126572 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.126621 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260928 06:33:30.127666 5316 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9bef5525643d2a96b815d355e8348 (DEFAULT_TABLE table=test-table [id=8d7726592a3b4efb9207493ab4c62b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:30.128046 5316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9bef5525643d2a96b815d355e8348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:30.128886 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.117333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.24.11.129:39459)
0928 06:33:30.117477 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.117481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.122477 (+ 4996us) server_negotiation.cc:407] Connection header received
0928 06:33:30.123915 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.123919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.123934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.123987 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:30.125053 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.126130 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.127415 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.128157 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.128206 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.128259 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.128711 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.128771 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 06:33:30.129021 5409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.117001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33203 (local address 127.0.0.1:33302)
0928 06:33:30.121731 (+ 4730us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.121742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:30.121803 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:30.122509 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:30.122510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:30.122512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:30.122564 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.122565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.124195 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.124195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.127821 (+ 3626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.127824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.128794 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.128795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.128863 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.128867 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:30.128912 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:33:30.128952 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"thread_start_us":92,"threads_started":1}
I20260928 06:33:30.129192 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.117233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39459 (local address 127.0.0.1:55162)
0928 06:33:30.122409 (+ 5176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.122419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:30.123814 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:30.124095 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:30.124095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:30.124097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:30.124141 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.124142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.126227 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.126228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.127314 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.127316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.128566 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.128567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.128620 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.128624 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:30.128992 (+ 368us) client_negotiation.cc:240] Negotiation successful
0928 06:33:30.129038 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"thread_start_us":80,"threads_started":1}
I20260928 06:33:30.129244 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.117096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.24.11.130:33203)
0928 06:33:30.121660 (+ 4564us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.121666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.121877 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:33:30.121916 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.121919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.121935 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.122281 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:30.122632 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.123721 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.127921 (+ 4200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.128641 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.128711 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.128761 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.129125 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.129173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4525,"thread_start_us":53,"threads_started":1}
I20260928 06:33:30.129927 5048 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9bef5525643d2a96b815d355e8348 (DEFAULT_TABLE table=test-table [id=8d7726592a3b4efb9207493ab4c62b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:30.130149 5182 tablet_service.cc:1467] Processing CreateTablet for tablet b1c9bef5525643d2a96b815d355e8348 (DEFAULT_TABLE table=test-table [id=8d7726592a3b4efb9207493ab4c62b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:30.130285 5048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9bef5525643d2a96b815d355e8348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:30.130446 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c9bef5525643d2a96b815d355e8348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:30.134032 5416 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Bootstrap starting.
I20260928 06:33:30.134671 5416 tablet_bootstrap.cc:654] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:30.134989 5416 log.cc:824] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:30.135351 5417 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: Bootstrap starting.
I20260928 06:33:30.135739 5416 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: No bootstrap required, opened a new log
I20260928 06:33:30.135828 5416 ts_tablet_manager.cc:1397] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:30.137275 5416 raft_consensus.cc:348] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.137394 5416 raft_consensus.cc:374] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:30.137415 5416 raft_consensus.cc:729] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918776aece0941d1826a0baaf27f05ba, State: Initialized, Role: FOLLOWER
I20260928 06:33:30.137565 5416 consensus_queue.cc:260] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.137822 5416 ts_tablet_manager.cc:1428] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:30.138060 5248 heartbeater.cc:503] Master 127.24.11.190:38131 was elected leader, sending a full tablet report...
I20260928 06:33:30.138329 5417 tablet_bootstrap.cc:654] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:30.138623 5417 log.cc:824] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:30.139230 5415 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Bootstrap starting.
I20260928 06:33:30.139387 5417 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: No bootstrap required, opened a new log
I20260928 06:33:30.139446 5417 ts_tablet_manager.cc:1397] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:33:30.139909 5415 tablet_bootstrap.cc:654] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:30.140236 5415 log.cc:824] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:30.141211 5415 tablet_bootstrap.cc:492] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: No bootstrap required, opened a new log
I20260928 06:33:30.141274 5415 ts_tablet_manager.cc:1397] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:30.142491 5417 raft_consensus.cc:348] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.142650 5417 raft_consensus.cc:374] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:30.142683 5417 raft_consensus.cc:729] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ef8ba2e53d456ebe884241e3d41de9, State: Initialized, Role: FOLLOWER
I20260928 06:33:30.142822 5417 consensus_queue.cc:260] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.143078 5417 ts_tablet_manager.cc:1428] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:33:30.143189 5114 heartbeater.cc:503] Master 127.24.11.190:38131 was elected leader, sending a full tablet report...
I20260928 06:33:30.144135 5415 raft_consensus.cc:348] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.144284 5415 raft_consensus.cc:374] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:30.144305 5415 raft_consensus.cc:729] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3c2b35a2f64cf189c86e51c1cb93f6, State: Initialized, Role: FOLLOWER
I20260928 06:33:30.144441 5415 consensus_queue.cc:260] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.144644 5415 ts_tablet_manager.cc:1428] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:30.144753 5382 heartbeater.cc:503] Master 127.24.11.190:38131 was elected leader, sending a full tablet report...
W20260928 06:33:30.153294 5249 tablet.cc:2369] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:30.175223 5423 raft_consensus.cc:482] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:30.175483 5423 raft_consensus.cc:504] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.175885 5423 leader_election.cc:288] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 918776aece0941d1826a0baaf27f05ba (127.24.11.130:33203), 49ef8ba2e53d456ebe884241e3d41de9 (127.24.11.129:39459)
I20260928 06:33:30.180092 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.176328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:48917 (local address 127.24.11.129:39459)
0928 06:33:30.176373 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.176375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.176508 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:33:30.176611 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.176612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.176621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.176651 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:30.176867 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.177631 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179149 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.179712 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179759 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.179768 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.179978 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.180018 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:33:30.180153 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.176347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:36599 (local address 127.24.11.130:33203)
0928 06:33:30.176389 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.176391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.176541 (+ 150us) server_negotiation.cc:407] Connection header received
0928 06:33:30.176645 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.176646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.176655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.176685 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:30.176892 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.177638 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179158 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.179735 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179787 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.179795 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.180039 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.180083 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:30.180408 5425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.176129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33203 (local address 127.24.11.131:36599)
0928 06:33:30.176473 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.176483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:30.176570 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:30.176749 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:30.176749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:30.176751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:30.176818 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.176820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.177741 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.177741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.179050 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.179053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.179854 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.179944 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.179948 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:30.180270 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 06:33:30.180316 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260928 06:33:30.180754 5424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.176088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39459 (local address 127.24.11.131:48917)
0928 06:33:30.176442 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.176453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:30.176538 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:30.176717 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:30.176718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:30.176720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:30.176797 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.176799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.177727 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.177728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.179047 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:30.179049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.179835 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.179836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:30.179902 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.179906 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:30.180530 (+ 624us) client_negotiation.cc:240] Negotiation successful
0928 06:33:30.180663 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":54,"thread_start_us":116,"threads_started":1}
I20260928 06:33:30.180837 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9bef5525643d2a96b815d355e8348" candidate_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918776aece0941d1826a0baaf27f05ba" is_pre_election: true
I20260928 06:33:30.181020 5202 raft_consensus.cc:2454] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c2b35a2f64cf189c86e51c1cb93f6 in term 0.
I20260928 06:33:30.181241 5272 leader_election.cc:302] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c2b35a2f64cf189c86e51c1cb93f6, 918776aece0941d1826a0baaf27f05ba; no voters:
I20260928 06:33:30.181205 5068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9bef5525643d2a96b815d355e8348" candidate_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef8ba2e53d456ebe884241e3d41de9" is_pre_election: true
I20260928 06:33:30.181355 5068 raft_consensus.cc:2454] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a3c2b35a2f64cf189c86e51c1cb93f6 in term 0.
I20260928 06:33:30.181624 5423 raft_consensus.cc:2781] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:30.181674 5423 raft_consensus.cc:482] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:30.181695 5423 raft_consensus.cc:3037] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:30.182664 5423 raft_consensus.cc:504] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.182796 5423 leader_election.cc:288] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [CANDIDATE]: Term 1 election: Requested vote from peers 918776aece0941d1826a0baaf27f05ba (127.24.11.130:33203), 49ef8ba2e53d456ebe884241e3d41de9 (127.24.11.129:39459)
I20260928 06:33:30.183079 5068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9bef5525643d2a96b815d355e8348" candidate_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49ef8ba2e53d456ebe884241e3d41de9"
I20260928 06:33:30.183177 5068 raft_consensus.cc:3037] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:30.183988 5068 raft_consensus.cc:2454] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c2b35a2f64cf189c86e51c1cb93f6 in term 1.
I20260928 06:33:30.184185 5270 leader_election.cc:302] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3c2b35a2f64cf189c86e51c1cb93f6, 49ef8ba2e53d456ebe884241e3d41de9; no voters:
I20260928 06:33:30.184281 5423 raft_consensus.cc:2781] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:30.184314 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c9bef5525643d2a96b815d355e8348" candidate_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "918776aece0941d1826a0baaf27f05ba"
I20260928 06:33:30.184386 5202 raft_consensus.cc:3037] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:30.184442 5423 raft_consensus.cc:686] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2a3c2b35a2f64cf189c86e51c1cb93f6, State: Running, Role: LEADER
I20260928 06:33:30.184538 5423 consensus_queue.cc:237] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } }
I20260928 06:33:30.185142 5202 raft_consensus.cc:2454] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a3c2b35a2f64cf189c86e51c1cb93f6 in term 1.
I20260928 06:33:30.185384 4915 catalog_manager.cc:5495] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a3c2b35a2f64cf189c86e51c1cb93f6 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3c2b35a2f64cf189c86e51c1cb93f6" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 } health_report { overall_health: UNKNOWN } } }
I20260928 06:33:30.194638 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.191348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.24.11.131:37001)
0928 06:33:30.191387 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:30.191389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:30.191399 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:30.191463 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:30.191464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:30.191483 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:30.191566 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:30.191697 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.192426 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.193551 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:30.193846 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:30.193909 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:30.194102 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:30.194238 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:30.194315 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:30.194541 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:33:30.194563 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:30.197956 5068 raft_consensus.cc:1261] T b1c9bef5525643d2a96b815d355e8348 P 49ef8ba2e53d456ebe884241e3d41de9 [term 1 FOLLOWER]: Refusing update from remote peer 2a3c2b35a2f64cf189c86e51c1cb93f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:30.198015 5202 raft_consensus.cc:1261] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 1 FOLLOWER]: Refusing update from remote peer 2a3c2b35a2f64cf189c86e51c1cb93f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:30.198405 5423 consensus_queue.cc:1035] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49ef8ba2e53d456ebe884241e3d41de9" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:30.198614 5426 consensus_queue.cc:1035] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "918776aece0941d1826a0baaf27f05ba" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:30.201735 5429 mvcc.cc:205] Tried to move back new op lower bound from 7334204252968214528 to 7334204252916461568. Current Snapshot: MvccSnapshot[applied={T|T < 7334204252968214528}]
I20260928 06:33:30.202296 5431 mvcc.cc:205] Tried to move back new op lower bound from 7334204252968214528 to 7334204252916461568. Current Snapshot: MvccSnapshot[applied={T|T < 7334204252968214528}]
I20260928 06:33:30.203024 5430 mvcc.cc:205] Tried to move back new op lower bound from 7334204252968214528 to 7334204252916461568. Current Snapshot: MvccSnapshot[applied={T|T < 7334204252968214528}]
I20260928 06:33:30.205176 4915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:30.205281 4915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:30.206065 4915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efe0193525444c7680b8ba0eae0d13b2: Sending DeleteTablet for 3 replicas of tablet b1c9bef5525643d2a96b815d355e8348
I20260928 06:33:30.206506 5316 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c9bef5525643d2a96b815d355e8348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:30 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56510
I20260928 06:33:30.206646 5048 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c9bef5525643d2a96b815d355e8348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:30 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:55162
I20260928 06:33:30.206892 5182 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c9bef5525643d2a96b815d355e8348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:30 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:33302
I20260928 06:33:30.207548 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4954
I20260928 06:33:30.217581 5438 tablet_replica.cc:330] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: stopping tablet replica
I20260928 06:33:30.217823 5438 raft_consensus.cc:2229] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:30.218117 5438 raft_consensus.cc:2258] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:30.218395 5440 tablet_replica.cc:330] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: stopping tablet replica
I20260928 06:33:30.218581 5440 raft_consensus.cc:2229] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:30.218760 5440 raft_consensus.cc:2258] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:30.218791 5438 ts_tablet_manager.cc:1905] T b1c9bef5525643d2a96b815d355e8348 P 2a3c2b35a2f64cf189c86e51c1cb93f6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:33:30.219250 4901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:39459 (error 108)
I20260928 06:33:30.219493 5440 ts_tablet_manager.cc:1905] T b1c9bef5525643d2a96b815d355e8348 P 918776aece0941d1826a0baaf27f05ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:30.220283 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5116
I20260928 06:33:30.220287 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.220135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39459 (local address 127.0.0.1:55164)
0928 06:33:30.220191 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.220242 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:33:30.226239 4901 catalog_manager.cc:4584] TS 49ef8ba2e53d456ebe884241e3d41de9 (127.24.11.129:39459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1c9bef5525643d2a96b815d355e8348: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39459: connect: Connection refused (error 111)
I20260928 06:33:30.228739 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5251
I20260928 06:33:30.230967 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:30.230829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33203 (local address 127.0.0.1:33316)
0928 06:33:30.230888 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:30.230917 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:33:30.234948 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 4883
I20260928 06:33:30.240196 24622 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:33:30.584844 24622 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:33:30.584928 24622 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:33:30.929441 24622 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:33:30.929 UTC [3950] LOG: received smart shutdown request
2026-09-28 06:33:30.933 UTC [3950] LOG: background worker "logical replication launcher" (PID 3959) exited with exit code 1
2026-09-28 06:33:30.934 UTC [3954] LOG: shutting down
2026-09-28 06:33:31.106 UTC [3950] LOG: database system is shut down
2026-09-28T06:33:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49138 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:31 dist-test-slave-38pg krb5kdc[5450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:31 dist-test-slave-38pg krb5kdc[5450](info): set up 2 sockets
Sep 28 06:33:31 dist-test-slave-38pg krb5kdc[5450](info): commencing operation
krb5kdc: starting...
W20260928 06:33:32.250540 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:33:32.372190 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35921
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:32.453514 5463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:32.453758 5463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:32.453781 5463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:32.455129 5463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:32.455183 5463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:32.455199 5463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:32.455212 5463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:32.455225 5463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:32.456939 5463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:32.457216 5463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:32.457473 5463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:32.462347 5463 server_base.cc:1034] running on GCE node
W20260928 06:33:32.462302 5471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.462358 5469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.462318 5468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:32.462920 5463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:32.463256 5463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:32.464418 5463 hybrid_clock.cc:648] HybridClock initialized: now 1790577212464396 us; error 44 us; skew 500 ppm
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:32.470948 5463 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:32.471462 5463 webserver.cc:460] Webserver started at http://127.24.11.190:42257/ using document root <none> and password file <none>
I20260928 06:33:32.471696 5463 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:32.471742 5463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:32.471887 5463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:32.472991 5463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "00798ec377e74803b2396edd06e6c9cc"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "45efba2df38204b6e9757140ba6e963d"
server_key_iv: "3e47a3aa85b8442d7ad1e7092e5bda48"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.473359 5463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "00798ec377e74803b2396edd06e6c9cc"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "45efba2df38204b6e9757140ba6e963d"
server_key_iv: "3e47a3aa85b8442d7ad1e7092e5bda48"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.474743 5463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:32.475530 5478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:32.475716 5463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:32.475824 5463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "00798ec377e74803b2396edd06e6c9cc"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "45efba2df38204b6e9757140ba6e963d"
server_key_iv: "3e47a3aa85b8442d7ad1e7092e5bda48"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.475925 5463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:32.518028 5463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:32.520112 5463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:32.520294 5463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:32.524442 5463 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:35921
I20260928 06:33:32.524482 5530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:35921 every 8 connection(s)
I20260928 06:33:32.524878 5463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:32.525852 5531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:32.527344 5531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.527747 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5463
I20260928 06:33:32.527856 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:33:32.528043 24622 external_mini_cluster.cc:1428] Setting key 6fc59007d9a82e9cc35f5b6a9044bc17
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.189@KRBTEST.COM; defaulting to no policy
I20260928 06:33:32.538319 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.528146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35921 (local address 127.0.0.1:33806)
0928 06:33:32.528344 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.528384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.528848 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.529169 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.529171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.529607 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.529848 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.529855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.530908 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.530910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.532102 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.532106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.532617 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.532618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.532738 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.533287 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.533299 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.534973 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.536789 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.536795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.536797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.537069 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.537296 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.537297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.537297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.537386 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.537871 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.537877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.537939 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.538036 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.538240 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":74,"thread_start_us":112,"threads_started":1}
I20260928 06:33:32.538326 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.528167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.24.11.190:35921)
0928 06:33:32.528383 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.528392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.528472 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:33:32.529004 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.529007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.529029 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.529139 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.529958 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.530765 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.532196 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.532516 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.532605 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.535074 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.535080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.535081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.535090 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.536663 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.537146 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.537149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.537150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.537228 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.537464 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.537465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.537466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.537733 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.537868 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.538060 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.538154 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.189@KRBTEST.COM" created.
I20260928 06:33:32.539068 5531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.539459 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.539319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:45974)
0928 06:33:32.539372 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.539424 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:33:32.539613 5479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
I20260928 06:33:32.540113 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.540007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:45986)
0928 06:33:32.540053 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.540084 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:33:32.540294 5531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:45371: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
I20260928 06:33:32.554445 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:45371
--webserver_interface=127.24.11.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:32.589598 5531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } attempt: 1
I20260928 06:33:32.590142 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.589957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:45996)
0928 06:33:32.590041 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.590096 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:33:32.590587 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.590458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46012)
0928 06:33:32.590517 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.590554 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:33:32.590698 5531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:45371: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
W20260928 06:33:32.635399 5538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:32.635624 5538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:32.635648 5538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:32.636849 5538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:32.636898 5538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:32.636912 5538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:32.636925 5538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:32.636979 5538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:32.638526 5538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.11.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:32.638792 5538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:32.639036 5538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:32.643851 5543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.643843 5546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.643843 5544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:32.644171 5538 server_base.cc:1034] running on GCE node
I20260928 06:33:32.644439 5538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:32.644690 5538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:32.645856 5538 hybrid_clock.cc:648] HybridClock initialized: now 1790577212645811 us; error 39 us; skew 500 ppm
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:32.648363 5538 init.cc:377] Logged in from keytab as kudu/127.24.11.189@KRBTEST.COM (short username kudu)
I20260928 06:33:32.648866 5538 webserver.cc:460] Webserver started at http://127.24.11.189:35049/ using document root <none> and password file <none>
I20260928 06:33:32.649098 5538 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:32.649168 5538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:32.649322 5538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:32.650503 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data/instance:
uuid: "983d9fcdb1594a728ba188f6c75e8ea7"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "42dd89e341e9d255ca30db3bd90901ab"
server_key_iv: "99b315240ebf4ace6bd0bec9a13ebb3c"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.650987 5538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal/instance:
uuid: "983d9fcdb1594a728ba188f6c75e8ea7"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "42dd89e341e9d255ca30db3bd90901ab"
server_key_iv: "99b315240ebf4ace6bd0bec9a13ebb3c"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.652398 5538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:32.653179 5553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:32.653478 5538 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:32.653616 5538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "983d9fcdb1594a728ba188f6c75e8ea7"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "42dd89e341e9d255ca30db3bd90901ab"
server_key_iv: "99b315240ebf4ace6bd0bec9a13ebb3c"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.653699 5538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:32.658843 5531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } attempt: 2
I20260928 06:33:32.659314 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.659181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46018)
0928 06:33:32.659240 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.659283 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:33:32.659662 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.659581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46020)
0928 06:33:32.659619 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.659646 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:33:32.659787 5531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:45371: Network error: Client connection negotiation failed: client connection to 127.24.11.189:45371: connect: Connection refused (error 111)
I20260928 06:33:32.668649 5538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:32.670639 5538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:32.670806 5538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:32.675076 5538 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.189:45371
I20260928 06:33:32.675128 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.189:45371 every 8 connection(s)
I20260928 06:33:32.675509 5538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
I20260928 06:33:32.676362 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:32.677896 5606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.679098 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5538
I20260928 06:33:32.679172 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-1/wal/instance
I20260928 06:33:32.679363 24622 external_mini_cluster.cc:1428] Setting key 68f7a3c96bc3f87fe01af111f3232b81
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.188@KRBTEST.COM; defaulting to no policy
I20260928 06:33:32.688001 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.678567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.24.11.190:35921)
0928 06:33:32.678630 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.678632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.678859 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:33:32.679385 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.679386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.679416 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.679514 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.680144 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.680925 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.682477 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.682689 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.682744 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.685179 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.685181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.685181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.685189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.686655 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.687106 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.687107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.687108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.687172 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.687413 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.687414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.687414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.687530 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.687587 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.687857 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.687931 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:33:32.688068 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.678568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35921 (local address 127.0.0.1:33812)
0928 06:33:32.678720 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.678763 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.679281 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.679598 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.679601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.679858 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.680056 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.680059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.681020 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.681021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.682396 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.682399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.682778 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.682779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.682951 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.683502 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.683514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.685086 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.686759 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.686766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.686769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.687025 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.687253 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.687254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.687254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.687351 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.687591 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.687595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.687705 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.687788 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.687910 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260928 06:33:32.688616 5606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.188@KRBTEST.COM" created.
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:33:32.696569 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.688868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46032)
0928 06:33:32.688919 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.688931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.688974 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.689244 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.689244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.689391 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.689456 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.689458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.690535 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.690536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.691045 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.691047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.691673 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.691674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.691756 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.692152 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.692158 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.693508 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.695593 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.695594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.695595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.695748 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.695972 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.695973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.695973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.696021 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.696339 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.696340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.696374 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.696428 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.696504 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:33:32.696699 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.688964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.24.11.189:45371)
0928 06:33:32.689106 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.689109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.689124 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:33:32.689140 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.689141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.689151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.689265 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.689553 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.690440 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.691132 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.691597 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.691655 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.693633 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.693642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.693642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.693650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.695512 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.695837 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.695840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.695841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.695907 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.696079 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.696080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.696081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.696269 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.696360 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.696467 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.696573 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 06:33:32.697330 5606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.697737 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.697596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45678)
0928 06:33:32.697658 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.697709 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 06:33:32.697893 5557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
I20260928 06:33:32.698334 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.698222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45694)
0928 06:33:32.698278 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.698309 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:33:32.698475 5606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:44819: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
I20260928 06:33:32.706017 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:44819
--webserver_interface=127.24.11.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:32.747750 5606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } attempt: 1
I20260928 06:33:32.748278 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.748144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45704)
0928 06:33:32.748206 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.748249 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:33:32.748631 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.748536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45718)
0928 06:33:32.748580 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.748607 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:33:32.748714 5606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:44819: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
I20260928 06:33:32.750962 5531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } attempt: 3
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:33:32.758307 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.751327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.24.11.189:45371)
0928 06:33:32.751394 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.751396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.751408 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:33:32.751448 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.751449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.751462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.751524 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.751893 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.752408 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.753104 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.753326 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.753404 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.755515 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.755517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.755517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.755525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.757060 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.757431 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.757432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.757433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.757490 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.757818 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.757818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.757819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.757951 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.758007 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.758171 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.758240 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:32.758387 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.751237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46048)
0928 06:33:32.751289 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.751303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.751385 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.751529 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.751530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.751694 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.751757 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.751759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.752490 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.752491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.753016 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.753017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.753402 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.753403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.753482 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.754017 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.754022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.755416 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.757157 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.757158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.757158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.757335 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.757662 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.757663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.757664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.757741 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.758019 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.758020 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.758071 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.758145 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.758247 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:33:32.759265 5531 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.759675 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.759553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45722)
0928 06:33:32.759610 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.759646 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:33:32.759977 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.759888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45724)
0928 06:33:32.759933 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.759957 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:33:32.760062 5531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:44819: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
W20260928 06:33:32.790678 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:32.790908 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:32.790932 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:33:32.791203 5531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } attempt: 1
I20260928 06:33:32.791651 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.791514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45734)
0928 06:33:32.791590 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.791622 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:33:32.791999 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.791906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45740)
0928 06:33:32.791949 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.791981 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:33:32.792084 5531 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:44819: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
W20260928 06:33:32.792399 5613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:32.792443 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:32.792459 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:32.792500 5613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:32.792532 5613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:32.794106 5613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:35921,127.24.11.189:45371,127.24.11.188:44819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.11.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:32.794358 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:32.794591 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:32.799594 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.799579 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:32.799574 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:32.799708 5613 server_base.cc:1034] running on GCE node
I20260928 06:33:32.800112 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:32.800415 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:32.801611 5613 hybrid_clock.cc:648] HybridClock initialized: now 1790577212801596 us; error 30 us; skew 500 ppm
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:32.804059 5613 init.cc:377] Logged in from keytab as kudu/127.24.11.188@KRBTEST.COM (short username kudu)
I20260928 06:33:32.804531 5613 webserver.cc:460] Webserver started at http://127.24.11.188:41087/ using document root <none> and password file <none>
I20260928 06:33:32.804724 5613 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:32.804766 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:32.804855 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:32.805943 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data/instance:
uuid: "7959587b570d4e6db55b855bb4260eaa"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "e61e8b7a0e899a78222d3100d4448c37"
server_key_iv: "e62fd14f7f29339dbf17fa3fe06e9f8a"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.806241 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal/instance:
uuid: "7959587b570d4e6db55b855bb4260eaa"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "e61e8b7a0e899a78222d3100d4448c37"
server_key_iv: "e62fd14f7f29339dbf17fa3fe06e9f8a"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.807484 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:32.808146 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:32.808311 5613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:32.808396 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "7959587b570d4e6db55b855bb4260eaa"
format_stamp: "Formatted at 2026-09-28 06:33:32 on dist-test-slave-38pg"
server_key: "e61e8b7a0e899a78222d3100d4448c37"
server_key_iv: "e62fd14f7f29339dbf17fa3fe06e9f8a"
server_key_version: "encryptionkey@0"
I20260928 06:33:32.808449 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:32.816857 5606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } attempt: 2
I20260928 06:33:32.817339 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.817183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45756)
0928 06:33:32.817248 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.817296 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:33:32.817723 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.817632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45760)
0928 06:33:32.817675 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.817702 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:33:32.817852 5606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:44819: Network error: Client connection negotiation failed: client connection to 127.24.11.188:44819: connect: Connection refused (error 111)
I20260928 06:33:32.819496 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:32.821756 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:32.821925 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:32.825793 5613 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.188:44819
I20260928 06:33:32.825826 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.188:44819 every 8 connection(s)
I20260928 06:33:32.826195 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
I20260928 06:33:32.827077 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:32.828565 5681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.830667 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5613
I20260928 06:33:32.830739 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577055356400-24622-0/minicluster-data/master-2/wal/instance
I20260928 06:33:32.830917 24622 external_mini_cluster.cc:1428] Setting key cc34a15024a3b05208071b2afe6ea61d
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:33:32.838272 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.829212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.24.11.190:35921)
0928 06:33:32.829298 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.829300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.829466 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:33:32.829982 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.829983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.829993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.830045 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.830651 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.831294 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.832756 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.832955 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.833035 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.835319 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.835320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.835321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.835335 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.836815 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.837274 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.837275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.837276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.837340 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.837617 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.837618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.837619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.837735 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.837813 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.838102 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.838182 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:33:32.838467 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.829174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:35921 (local address 127.0.0.1:33816)
0928 06:33:32.829351 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.829387 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.829896 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.830109 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.830112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.830367 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.830553 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.830557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.831452 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.831453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.832669 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.832672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.833040 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.833041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.833195 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.833763 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.833772 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.835226 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.836900 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.836907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.836910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.837188 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.837412 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.837413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.837413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.837524 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.837804 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.837811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.837959 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.838068 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.838220 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260928 06:33:32.839061 5681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.840283 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.831301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.24.11.188:44819)
0928 06:33:32.831462 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.831468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.831526 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:33:32.831599 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.831601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.831617 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.831678 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.832412 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.833088 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.834092 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.834376 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.834462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.836969 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.836976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.836977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.836986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.838889 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.839271 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.839274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.839274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.839337 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.839637 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.839638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.839638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.839782 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.839953 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.840053 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.840148 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:32.845602 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.839321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:45371 (local address 127.0.0.1:46052)
0928 06:33:32.839376 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.839386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.839420 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.839571 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.839572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.839677 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.839708 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.839709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.840410 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.840411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.841091 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.841094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.841447 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.841448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.841505 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.841933 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.841938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.843032 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.844536 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.844537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.844538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.844753 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.844974 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.844974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.844974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.845024 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.845293 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.845294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.845328 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.845377 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.845486 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:33:32.845638 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.839415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.24.11.189:45371)
0928 06:33:32.839461 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.839463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.839471 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:32.839496 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.839497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.839507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.839567 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.839784 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.840333 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.841156 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.841361 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.841449 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.843107 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.843109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.843109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.843117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.844454 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.844842 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.844843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.844844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.844904 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.845088 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.845089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.845090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.845228 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.845298 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.845430 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.845516 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:32.845986 5681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:33:32.848909 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.841088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.24.11.190:35921)
0928 06:33:32.841125 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.841127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.841177 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:33:32.841249 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.841250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.841258 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.841312 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.841853 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.842287 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.842984 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.843274 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.843351 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.846078 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.846079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.846080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.846087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.847766 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.848101 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.848103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.848103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.848163 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.848351 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.848352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.848352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.848448 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.848613 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.848693 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.848774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:33:32.853477 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.846299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.24.11.188:44819)
0928 06:33:32.846332 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.846333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.846340 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:32.846398 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.846398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.846406 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.846447 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.846737 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.847345 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.848212 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.848513 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.848574 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.850952 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.850954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.850954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.850960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.852348 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.852753 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.852754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.852755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.852803 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.853001 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.853002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.853002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.853093 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.853141 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.853307 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.853414 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":16}
I20260928 06:33:32.853652 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.846231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45774)
0928 06:33:32.846284 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.846295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.846334 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.846453 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.846454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.846599 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.846654 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.846656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.847427 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.847428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.848122 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.848124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.848594 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.848594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.848655 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.849170 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.849176 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.850873 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.852439 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.852440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.852440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.852679 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.852869 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.852870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.852870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.852937 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.853154 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.853155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.853206 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.853293 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.853479 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":53}
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:33:32.855259 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: Bootstrap starting.
I20260928 06:33:32.856163 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:32.856503 5681 log.cc:824] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:32.857379 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: No bootstrap required, opened a new log
I20260928 06:33:32.857705 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.849892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.24.11.189:45371)
0928 06:33:32.849927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.849929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.849952 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:33:32.849971 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.849974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.849987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.850026 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.850645 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.851246 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.851917 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.852113 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.852197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.854841 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.854842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.854842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.854850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.856438 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.856822 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.856823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.856823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.856893 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.857119 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.857120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.857120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.857240 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.857458 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.857496 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.857592 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:33:32.859319 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.859468 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:32.859490 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:32.859511 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7959587b570d4e6db55b855bb4260eaa, State: Initialized, Role: FOLLOWER
I20260928 06:33:32.859633 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.859975 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:32.859980 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:32.860102 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:32.861922 5704 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:32.861974 5704 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:32.862321 5613 master_runner.cc:186] Error getting master registration for 127.24.11.190:35921: OK, instance_id { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" instance_seqno: 1790577212523239 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:32.864660 5613 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:33:32.867205 5531 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } attempt: 2
I20260928 06:33:32.869029 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.860997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.24.11.188:44819)
0928 06:33:32.861066 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.861068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.861169 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:33:32.861231 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.861232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.861243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.861309 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.862190 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.862864 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.863870 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.864149 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.864209 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.865642 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.865643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.865643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.865664 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.867613 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.868027 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.868028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.868029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.868090 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.868281 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.868282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.868282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.868428 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.868546 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.868745 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.868937 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:32.870172 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.861048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.24.11.190:35921)
0928 06:33:32.861092 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.861094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.861129 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:33:32.861241 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.861242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.861255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.861289 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.862125 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.862840 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.864171 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.864446 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.864806 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.866710 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.866711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.866712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.866719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.868386 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.868771 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.868772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.868773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.868839 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.869056 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.869056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.869057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.869189 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.869261 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.870007 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.870084 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:33:32.877121 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.863256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.24.11.189:45371)
0928 06:33:32.863305 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.863307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.868891 (+ 5584us) server_negotiation.cc:407] Connection header received
0928 06:33:32.868995 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.868997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.869009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.869042 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.869625 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.870319 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.871270 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.871567 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.871643 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.872927 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.872928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.872929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.872935 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.874728 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.876150 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.876152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.876152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.876213 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.876432 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.876433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.876433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.876571 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.876670 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.876934 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.877027 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:33:32.879933 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.867812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.24.11.188:44819)
0928 06:33:32.869486 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.869488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.869495 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:33:32.869511 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.869512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.869520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.869900 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.869929 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.870621 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.871484 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.871747 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.871816 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.873820 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.873821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.873821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.873828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.878614 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.879042 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.879043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.879043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.879106 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.879327 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.879328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.879334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.879481 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.879552 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.879758 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.879841 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1661,"thread_start_us":101,"threads_started":1}
I20260928 06:33:32.879979 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.867960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45802)
0928 06:33:32.868012 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.868023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.868061 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.869637 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.869638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.869791 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.869831 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.869833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.870696 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.870697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.871405 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.871406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.871827 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.871828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.871886 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.872281 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.872286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.873747 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.878715 (+ 4968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.878717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.878717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.878955 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.879192 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.879193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.879193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.879274 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.879570 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.879571 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.879639 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.879721 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.879820 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:33:32.882776 5531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: Bootstrap starting.
I20260928 06:33:32.883579 5531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:32.883912 5531 log.cc:824] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:32.884729 5531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: No bootstrap required, opened a new log
I20260928 06:33:32.886554 5531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.886689 5531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:32.886734 5531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00798ec377e74803b2396edd06e6c9cc, State: Initialized, Role: FOLLOWER
I20260928 06:33:32.886871 5531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.887173 5531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:32.887152 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:32.887243 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:32.888935 5720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:32.888998 5720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:32.889709 5463 master_runner.cc:186] Error getting master registration for 127.24.11.189:45371: OK, instance_id { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" instance_seqno: 1790577212673851 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:33:32.892566 5463 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:33:32.909047 5606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } attempt: 3
Sep 28 06:33:32 dist-test-slave-38pg krb5kdc[5450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577212, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:33:32.918120 5608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.909397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:44819 (local address 127.0.0.1:45812)
0928 06:33:32.909456 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:32.909467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:32.909507 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:32.909722 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:32.909723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:32.909920 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:32.909994 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.909997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.911009 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.911011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.911910 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:32.911912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.912857 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.912858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:32.912924 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.913473 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:32.913479 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:32.915079 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:32.916985 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.916986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.916987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.917211 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.917430 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:32.917431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:32.917432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:32.917498 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:32.917796 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:32.917797 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:32.917847 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 06:33:32.917912 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:33:32.918013 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:33:32.918143 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:32.909502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.24.11.188:44819)
0928 06:33:32.909583 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:32.909586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:32.909595 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:32.909621 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:32.909622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:32.909632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:32.909699 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:32.910081 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.910915 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.912472 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:32.912779 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:32.912849 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:32.915178 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:32.915179 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:32.915180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:32.915188 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:32.916894 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.917290 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.917291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.917291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.917354 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:32.917585 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:32.917585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:32.917586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:32.917725 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:32.917794 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:32.917969 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:33:32.918050 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 06:33:32.920110 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: Bootstrap starting.
I20260928 06:33:32.920840 5606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:32.921141 5606 log.cc:824] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:32.922109 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: No bootstrap required, opened a new log
I20260928 06:33:32.924757 5606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.924904 5606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:32.924940 5606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983d9fcdb1594a728ba188f6c75e8ea7, State: Initialized, Role: FOLLOWER
I20260928 06:33:32.925078 5606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:32.925359 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:32.925414 5606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:32.926064 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:33:32.927361 5734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:32.927413 5734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:32.929404 5538 master_runner.cc:263] No leader master found from master 983d9fcdb1594a728ba188f6c75e8ea7
I20260928 06:33:32.929447 5538 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 06:33:33.021119 5484 tablet.cc:2369] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:33:33.072562 5634 tablet.cc:2369] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:33.130334 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:33.130467 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:33.130779 5709 leader_election.cc:288] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983d9fcdb1594a728ba188f6c75e8ea7 (127.24.11.189:45371), 7959587b570d4e6db55b855bb4260eaa (127.24.11.188:44819)
I20260928 06:33:33.131121 5655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00798ec377e74803b2396edd06e6c9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7959587b570d4e6db55b855bb4260eaa" is_pre_election: true
I20260928 06:33:33.131179 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00798ec377e74803b2396edd06e6c9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" is_pre_election: true
I20260928 06:33:33.131264 5655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00798ec377e74803b2396edd06e6c9cc in term 0.
I20260928 06:33:33.131359 5580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00798ec377e74803b2396edd06e6c9cc in term 0.
I20260928 06:33:33.131520 5482 leader_election.cc:302] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00798ec377e74803b2396edd06e6c9cc, 7959587b570d4e6db55b855bb4260eaa; no voters:
I20260928 06:33:33.131660 5709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:33.131717 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:33.131752 5709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:33.132565 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:33.132709 5709 leader_election.cc:288] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [CANDIDATE]: Term 1 election: Requested vote from peers 983d9fcdb1594a728ba188f6c75e8ea7 (127.24.11.189:45371), 7959587b570d4e6db55b855bb4260eaa (127.24.11.188:44819)
I20260928 06:33:33.132830 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00798ec377e74803b2396edd06e6c9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d9fcdb1594a728ba188f6c75e8ea7"
I20260928 06:33:33.132907 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:33.132920 5655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "00798ec377e74803b2396edd06e6c9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7959587b570d4e6db55b855bb4260eaa"
I20260928 06:33:33.132994 5655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:33.133791 5580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00798ec377e74803b2396edd06e6c9cc in term 1.
I20260928 06:33:33.133912 5655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00798ec377e74803b2396edd06e6c9cc in term 1.
I20260928 06:33:33.133975 5479 leader_election.cc:302] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00798ec377e74803b2396edd06e6c9cc, 983d9fcdb1594a728ba188f6c75e8ea7; no voters:
I20260928 06:33:33.134060 5709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:33.134222 5709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [term 1 LEADER]: Becoming Leader. State: Replica: 00798ec377e74803b2396edd06e6c9cc, State: Running, Role: LEADER
I20260928 06:33:33.134318 5709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } }
I20260928 06:33:33.134753 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00798ec377e74803b2396edd06e6c9cc. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.134855 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:33.135042 5738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:33.135231 5738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:33.136620 5580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [term 1 FOLLOWER]: Refusing update from remote peer 00798ec377e74803b2396edd06e6c9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:33.136633 5655 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [term 1 FOLLOWER]: Refusing update from remote peer 00798ec377e74803b2396edd06e6c9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:33.136963 5709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:33.137075 5736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:33.138357 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00798ec377e74803b2396edd06e6c9cc. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.138361 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00798ec377e74803b2396edd06e6c9cc. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.138444 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:33.138446 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:33.139381 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.139501 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:33.139477 5709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.139503 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.139566 5709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:33.139569 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:33.139864 5723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00798ec377e74803b2396edd06e6c9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00798ec377e74803b2396edd06e6c9cc" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 35921 } } peers { permanent_uuid: "983d9fcdb1594a728ba188f6c75e8ea7" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 45371 } } peers { permanent_uuid: "7959587b570d4e6db55b855bb4260eaa" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 44819 } } }
I20260928 06:33:33.139964 5723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:33:33.140460 5738 catalog_manager.cc:1347] Generated new cluster ID: d2b5866b12d34d4f9d7f460c554598e8
I20260928 06:33:33.140513 5738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:33.152734 5738 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:33.153591 5738 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:33.157352 5738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00798ec377e74803b2396edd06e6c9cc: Generated new TSK 0
I20260928 06:33:33.157634 5738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:33.862425 5704 catalog_manager.cc:1259] Loaded cluster ID: d2b5866b12d34d4f9d7f460c554598e8
I20260928 06:33:33.862498 5704 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: loading cluster ID for follower catalog manager: success
I20260928 06:33:33.863250 5704 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: acquiring CA information for follower catalog manager: success
I20260928 06:33:33.863505 5704 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7959587b570d4e6db55b855bb4260eaa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:33.927879 5734 catalog_manager.cc:1259] Loaded cluster ID: d2b5866b12d34d4f9d7f460c554598e8
I20260928 06:33:33.927989 5734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: loading cluster ID for follower catalog manager: success
I20260928 06:33:33.928797 5734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: acquiring CA information for follower catalog manager: success
I20260928 06:33:33.929119 5734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 983d9fcdb1594a728ba188f6c75e8ea7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:33:34.228404 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5463
I20260928 06:33:34.233636 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5538
I20260928 06:33:34.238598 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5613
2026-09-28T06:33:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3018 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:34 dist-test-slave-38pg krb5kdc[5753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:34 dist-test-slave-38pg krb5kdc[5753](info): set up 2 sockets
Sep 28 06:33:34 dist-test-slave-38pg krb5kdc[5753](info): commencing operation
krb5kdc: starting...
W20260928 06:33:35.274827 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:33:35.396687 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43963
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:35.476313 5766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:35.476526 5766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:35.476549 5766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:35.477715 5766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:35.477756 5766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:35.477770 5766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:35.477782 5766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:35.477795 5766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:35.479264 5766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:35.479533 5766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:35.479768 5766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:35.484789 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:35.484822 5766 server_base.cc:1034] running on GCE node
W20260928 06:33:35.484774 5774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:35.484752 5771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:35.485225 5766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:35.485605 5766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:35.486789 5766 hybrid_clock.cc:648] HybridClock initialized: now 1790577215486765 us; error 36 us; skew 500 ppm
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:35.489269 5766 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:35.489781 5766 webserver.cc:460] Webserver started at http://127.24.11.190:43219/ using document root <none> and password file <none>
I20260928 06:33:35.489979 5766 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:35.490020 5766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:35.490109 5766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:35.491155 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "414b36b6b4d34f018fdb7fe5469e8ffd"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-38pg"
server_key: "127fba4f14e23db840e232302ab1354e"
server_key_iv: "3c01eb103ae837b5e6b145b292126884"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.491466 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "414b36b6b4d34f018fdb7fe5469e8ffd"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-38pg"
server_key: "127fba4f14e23db840e232302ab1354e"
server_key_iv: "3c01eb103ae837b5e6b145b292126884"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.492722 5766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:33:35.493405 5781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:35.493645 5766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:35.493721 5766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "414b36b6b4d34f018fdb7fe5469e8ffd"
format_stamp: "Formatted at 2026-09-28 06:33:35 on dist-test-slave-38pg"
server_key: "127fba4f14e23db840e232302ab1354e"
server_key_iv: "3c01eb103ae837b5e6b145b292126884"
server_key_version: "encryptionkey@0"
I20260928 06:33:35.493777 5766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:35.507702 5766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:35.509670 5766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:35.509788 5766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:35.513587 5766 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43963
I20260928 06:33:35.513628 5833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43963 every 8 connection(s)
I20260928 06:33:35.513993 5766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:35.514889 5834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:35.517140 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd: Bootstrap starting.
I20260928 06:33:35.517786 5834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:35.518085 5834 log.cc:824] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:35.518777 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd: No bootstrap required, opened a new log
I20260928 06:33:35.520393 5834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } }
I20260928 06:33:35.520530 5834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:35.520553 5834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414b36b6b4d34f018fdb7fe5469e8ffd, State: Initialized, Role: FOLLOWER
I20260928 06:33:35.520720 5834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } }
I20260928 06:33:35.520781 5834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:35.520846 5834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:35.520886 5834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:35.521396 5834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } }
I20260928 06:33:35.521512 5834 leader_election.cc:302] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 414b36b6b4d34f018fdb7fe5469e8ffd; no voters:
I20260928 06:33:35.521697 5834 leader_election.cc:288] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:35.521690 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5766
I20260928 06:33:35.521783 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:33:35.521770 5837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:35.521983 24622 external_mini_cluster.cc:1428] Setting key 385590653ec817926ac8181a009b1f64
I20260928 06:33:35.521982 5834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:35.522013 5837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [term 1 LEADER]: Becoming Leader. State: Replica: 414b36b6b4d34f018fdb7fe5469e8ffd, State: Running, Role: LEADER
I20260928 06:33:35.522101 5837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } }
I20260928 06:33:35.522632 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } } }
I20260928 06:33:35.522716 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:35.522701 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 414b36b6b4d34f018fdb7fe5469e8ffd. Latest consensus state: current_term: 1 leader_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "414b36b6b4d34f018fdb7fe5469e8ffd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43963 } } }
I20260928 06:33:35.522961 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:35.523142 5848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:35.523720 5848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:35.525445 5848 catalog_manager.cc:1347] Generated new cluster ID: 3792c46a8b604acf8634041d4a39ae6c
I20260928 06:33:35.525494 5848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:35.532485 5846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:35.522530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.24.11.190:43963)
0928 06:33:35.522814 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:35.522824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:35.522861 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:33:35.522985 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:35.522993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:35.523053 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:35.523222 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:35.524251 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.525443 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.526242 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:35.526672 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:35.526802 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:35.529111 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:35.529116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:35.529116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:35.529129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:35.530886 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.531269 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.531273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.531273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.531383 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:35.531593 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:35.531594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:35.531594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:35.531828 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:35.531997 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:35.532068 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:33:35.532312 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":90,"threads_started":1}
I20260928 06:33:35.533388 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:33:35.544142 5848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:35.544667 5848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:35.552676 5848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 414b36b6b4d34f018fdb7fe5469e8ffd: Generated new TSK 0
I20260928 06:33:35.552889 5848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:33:35.587540 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5766
2026-09-28T06:33:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5861](info): set up 2 sockets
Sep 28 06:33:35 dist-test-slave-38pg krb5kdc[5861](info): commencing operation
krb5kdc: starting...
W20260928 06:33:36.614804 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:33:36 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:33:36.732997 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:36.813947 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:36.814160 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:36.814182 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:36.815642 5874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:36.815704 5874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:36.815721 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:36.815733 5874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:36.815745 5874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:36.817421 5874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:36.817718 5874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:36.817976 5874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:36.822883 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.822927 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:36.822923 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:36.823181 5874 server_base.cc:1034] running on GCE node
I20260928 06:33:36.823491 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:36.823848 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:36.825008 5874 hybrid_clock.cc:648] HybridClock initialized: now 1790577216824986 us; error 40 us; skew 500 ppm
Sep 28 06:33:36 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:36.827626 5874 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:36.828122 5874 webserver.cc:460] Webserver started at http://127.24.11.190:40967/ using document root <none> and password file <none>
I20260928 06:33:36.828370 5874 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:36.828415 5874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:36.828583 5874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:36.829808 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "28ea00026d4749a2be2bec16aacedc95"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-38pg"
server_key: "0210981929075a00b8f2af5c5a5c9925"
server_key_iv: "d9f9f8def0c6f5cb10c7e03ddee03c10"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.830184 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "28ea00026d4749a2be2bec16aacedc95"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-38pg"
server_key: "0210981929075a00b8f2af5c5a5c9925"
server_key_iv: "d9f9f8def0c6f5cb10c7e03ddee03c10"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.831547 5874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:36.832487 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:36.832711 5874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:36.832806 5874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "28ea00026d4749a2be2bec16aacedc95"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-38pg"
server_key: "0210981929075a00b8f2af5c5a5c9925"
server_key_iv: "d9f9f8def0c6f5cb10c7e03ddee03c10"
server_key_version: "encryptionkey@0"
I20260928 06:33:36.832903 5874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:36.858846 5874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:36.861049 5874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:36.861215 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:36.865357 5874 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:42751
I20260928 06:33:36.865491 5941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:42751 every 8 connection(s)
I20260928 06:33:36.865818 5874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:36.866873 5942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:36.868142 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5874
I20260928 06:33:36.868247 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:33:36.868458 24622 external_mini_cluster.cc:1428] Setting key 283ab233032d702a92d885767076b30f
I20260928 06:33:36.869917 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap starting.
I20260928 06:33:36.870728 5942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:36.871037 5942 log.cc:824] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:36.871742 5942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: No bootstrap required, opened a new log
I20260928 06:33:36.874182 5942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:36.874369 5942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:36.874411 5942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Initialized, Role: FOLLOWER
I20260928 06:33:36.874562 5942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:36.874665 5942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:36.874722 5942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:36.874773 5942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:36.875504 5942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:36.875594 5942 leader_election.cc:302] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28ea00026d4749a2be2bec16aacedc95; no voters:
Sep 28 06:33:36 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:36.875790 5942 leader_election.cc:288] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:36.875900 5947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:36.876083 5942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:36.876102 5947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 LEADER]: Becoming Leader. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Running, Role: LEADER
I20260928 06:33:36.876279 5947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:36.876729 5948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:36.876827 5948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:36.877715 5949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28ea00026d4749a2be2bec16aacedc95. Latest consensus state: current_term: 1 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:36.878087 5949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
W20260928 06:33:36.878136 5962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:36.878248 5962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:36.878355 5956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:36.878556 5956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:36.880470 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:36.869026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.24.11.190:42751)
0928 06:33:36.869204 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:36.869217 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:36.869242 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:33:36.869740 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:36.869747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:36.869797 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:36.869919 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:36.870713 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.871718 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.872806 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:36.873270 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:36.873362 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:36.876266 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:36.876270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:36.876270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:36.876284 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:36.878734 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.879184 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.879187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.879188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.879284 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:36.879513 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:36.879514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:36.879514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:36.879809 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:36.879932 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:36.880113 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:33:36.880272 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260928 06:33:36.880594 5956 catalog_manager.cc:1347] Generated new cluster ID: 4db13cc4b61d4ab18ddcb8fa618db0f8
I20260928 06:33:36.880635 5956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:36.888325 5956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:36.888867 5956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:36.892480 5956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Generated new TSK 0
I20260928 06:33:36.892678 5956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:33:36.922823 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:37.009016 5968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.009241 5968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.009268 5968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.010540 5968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.010588 5968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.010617 5968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:37.012065 5968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:37.012270 5968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.012454 5968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:37.017786 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.017789 5974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.018055 5968 server_base.cc:1034] running on GCE node
W20260928 06:33:37.017779 5973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.018334 5968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.018573 5968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.019737 5968 hybrid_clock.cc:648] HybridClock initialized: now 1790577217019729 us; error 40 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.022387 5968 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:37.022822 5968 webserver.cc:460] Webserver started at http://127.24.11.129:38781/ using document root <none> and password file <none>
I20260928 06:33:37.023059 5968 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.023130 5968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.023285 5968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:37.024374 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "39125b0dd79345ed9279d17c9f558ac8"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "3bafddd48bcc53dc1d4126414970ccc2"
server_key_iv: "d332344ec4932f980b0ada3bb2037ff3"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.024737 5968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "39125b0dd79345ed9279d17c9f558ac8"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "3bafddd48bcc53dc1d4126414970ccc2"
server_key_iv: "d332344ec4932f980b0ada3bb2037ff3"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.026204 5968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:37.027031 5983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.027201 5968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:37.027308 5968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "39125b0dd79345ed9279d17c9f558ac8"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "3bafddd48bcc53dc1d4126414970ccc2"
server_key_iv: "d332344ec4932f980b0ada3bb2037ff3"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.027384 5968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.041577 5968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.043633 5968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.043812 5968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.044063 5968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:37.044381 5968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:37.044443 5968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.044479 5968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:37.044508 5968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.050561 5968 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39625
I20260928 06:33:37.050645 6096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39625 every 8 connection(s)
I20260928 06:33:37.050987 5968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:37.058049 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 5968
I20260928 06:33:37.058167 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:37.058351 24622 external_mini_cluster.cc:1428] Setting key 1185f7fea1e679f6376b0c6b635ae6e8
I20260928 06:33:37.061058 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.051763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:59669)
0928 06:33:37.051930 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.051973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.052472 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.052716 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.052719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.053302 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:37.053514 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.053519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.054253 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.054254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.055099 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.055101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.055493 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.055494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.055597 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.056156 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:37.056165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:37.057608 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:37.059481 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.059490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.059492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.059835 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.060038 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.060039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.060039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.060195 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.060428 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:37.060434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:37.060618 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.060742 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.060885 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 06:33:37.061057 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.051887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:59669 (local address 127.24.11.190:42751)
0928 06:33:37.051953 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.051955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.052060 (+ 105us) server_negotiation.cc:407] Connection header received
0928 06:33:37.052567 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.052568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.052579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.052659 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.053637 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.054157 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.055189 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.055413 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.055471 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.057692 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.057693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.057693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.057701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.059402 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.059919 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.059920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.059920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.059972 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.060259 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.060259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.060259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.060362 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.060410 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.060783 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.060932 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:33:37.062048 6097 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:37.062165 6097 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:37.062366 6097 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:37.062984 5906 ts_manager.cc:194] Registered new tserver with Master: 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625)
I20260928 06:33:37.063664 5906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:59669
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:33:37.084479 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:37.166249 6102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.166465 6102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.166491 6102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.167586 6102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.167619 6102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.167645 6102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:37.169004 6102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:37.169185 6102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.169359 6102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:37.174248 6108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.174208 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.174208 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.174437 6102 server_base.cc:1034] running on GCE node
I20260928 06:33:37.174789 6102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.175060 6102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.176227 6102 hybrid_clock.cc:648] HybridClock initialized: now 1790577217176212 us; error 42 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.179039 6102 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:37.179458 6102 webserver.cc:460] Webserver started at http://127.24.11.130:42281/ using document root <none> and password file <none>
I20260928 06:33:37.179641 6102 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.179682 6102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.179792 6102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:37.180815 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "f2878ceab4bb467cafddb7223a4e709c"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "1a3b15a50e2c1abffdb6472fbfe8804b"
server_key_iv: "07c605c582751785cb16101a3812d170"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.181114 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "f2878ceab4bb467cafddb7223a4e709c"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "1a3b15a50e2c1abffdb6472fbfe8804b"
server_key_iv: "07c605c582751785cb16101a3812d170"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.182456 6102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:37.183169 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.183339 6102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:37.183429 6102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f2878ceab4bb467cafddb7223a4e709c"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "1a3b15a50e2c1abffdb6472fbfe8804b"
server_key_iv: "07c605c582751785cb16101a3812d170"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.183485 6102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.207700 6102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.209911 6102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.210031 6102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.210224 6102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:37.210585 6102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:37.210621 6102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.210644 6102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:37.210695 6102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.216580 6102 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37613
I20260928 06:33:37.216670 6230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37613 every 8 connection(s)
I20260928 06:33:37.217000 6102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:37.219583 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6102
I20260928 06:33:37.219674 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:37.219892 24622 external_mini_cluster.cc:1428] Setting key 30113f8f24063095d79c6d0595c2aa61
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:37.228255 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.217839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:59429 (local address 127.24.11.190:42751)
0928 06:33:37.217912 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.217914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.218161 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:33:37.218682 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.218683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.218695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.218763 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.219959 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.220666 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.222160 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.222499 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.222562 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.225224 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.225226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.225226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.225234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.226707 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.227260 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.227261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.227262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.227312 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.227602 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.227602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.227603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.227708 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.227758 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.228111 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.228182 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:37.228569 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.217829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:59429)
0928 06:33:37.218033 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.218071 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.218567 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.218843 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.218847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.219532 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:37.219837 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.219844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.220793 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.220794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.222056 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.222058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.222596 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.222597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.222755 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.223407 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:37.223415 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:37.225126 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:37.226803 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.226813 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.226817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.227180 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.227381 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.227382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.227383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.227534 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.227782 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:37.227789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:37.227967 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.228096 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.228284 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260928 06:33:37.229074 6231 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:37.229192 6231 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:37.229372 6231 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:37.229841 5906 ts_manager.cc:194] Registered new tserver with Master: f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
I20260928 06:33:37.230315 5906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:59429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:33:37.248790 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:37.331602 6236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.331856 6236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.331910 6236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.333338 6236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.333405 6236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.333451 6236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:37.335054 6236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:37.335341 6236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.335603 6236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:37.340976 6236 server_base.cc:1034] running on GCE node
W20260928 06:33:37.340946 6242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.340939 6241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.340932 6244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.341495 6236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.341790 6236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.342939 6236 hybrid_clock.cc:648] HybridClock initialized: now 1790577217342927 us; error 29 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.345652 6236 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:37.346105 6236 webserver.cc:460] Webserver started at http://127.24.11.131:40287/ using document root <none> and password file <none>
I20260928 06:33:37.346318 6236 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.346360 6236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.346452 6236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:37.347494 6236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "db2c874a4df04beab836e8db1a1997c0"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "19a66e16dac034efa692dea8fb21d26b"
server_key_iv: "44fb4d9a1436249e8a0499c58aec3425"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.347800 6236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "db2c874a4df04beab836e8db1a1997c0"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "19a66e16dac034efa692dea8fb21d26b"
server_key_iv: "44fb4d9a1436249e8a0499c58aec3425"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.349059 6236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:37.349882 6251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.350039 6236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:37.350129 6236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "db2c874a4df04beab836e8db1a1997c0"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "19a66e16dac034efa692dea8fb21d26b"
server_key_iv: "44fb4d9a1436249e8a0499c58aec3425"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.350184 6236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.364250 6236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.366832 6236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.366961 6236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.367148 6236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:37.367491 6236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:37.367532 6236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.367553 6236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:37.367566 6236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.373724 6236 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37215
I20260928 06:33:37.373792 6364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37215 every 8 connection(s)
I20260928 06:33:37.374115 6236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:37.383915 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.374862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:36349 (local address 127.24.11.190:42751)
0928 06:33:37.374945 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.374947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.375120 (+ 173us) server_negotiation.cc:407] Connection header received
0928 06:33:37.375594 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.375595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.375606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.375659 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.376668 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.377327 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.378577 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.378763 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.378813 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.381079 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.381080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.381081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.381087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.382527 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.382983 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.382984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.382984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.383043 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.383295 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.383296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.383296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.383431 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.383492 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.383782 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.383848 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:33:37.384107 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6236
I20260928 06:33:37.384198 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.374813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:36349)
0928 06:33:37.374998 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.375034 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.375513 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.375712 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.375716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.376353 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:37.376578 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.376583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.377433 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.377434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.378493 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.378495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.378835 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.378836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.378934 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.379518 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:37.379526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:37.380986 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:37.382608 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.382615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.382616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.382893 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.383113 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:37.383114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:37.383114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:37.383214 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:37.383518 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:37.383523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:37.383647 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.383749 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.383878 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260928 06:33:37.384210 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:37.384413 24622 external_mini_cluster.cc:1428] Setting key 338c443cf0ea1ec58cb8f482d10bf841
I20260928 06:33:37.384876 6365 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:37.384984 6365 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:37.385190 6365 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:37.385596 5906 ts_manager.cc:194] Registered new tserver with Master: db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215)
I20260928 06:33:37.385944 5906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:36349
I20260928 06:33:37.396224 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:37.403769 5945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.397651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.24.11.190:42751)
0928 06:33:37.397716 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.397718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.397819 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:33:37.397903 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.397904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.397916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.397988 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:37.398591 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.399092 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.399817 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.400008 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.400095 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.401293 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:37.401295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:37.401295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:37.401303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:37.402606 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.402962 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.402963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.402963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.403041 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:37.403219 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:37.403220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:37.403220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:37.403368 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:37.403462 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.403564 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.403635 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:33:37.405587 5906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:37.407122 5906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:37.413583 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.24.11.131:37215)
0928 06:33:37.409788 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.409792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.409873 (+ 81us) server_negotiation.cc:407] Connection header received
0928 06:33:37.409951 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.409952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.409964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.410059 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:37.410287 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411142 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.412525 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413162 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413251 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413312 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.413420 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.413462 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260928 06:33:37.413566 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:37215 (local address 127.0.0.1:59646)
0928 06:33:37.409808 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.409821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.409889 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.410127 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.410131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.410134 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:37.410218 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.410220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411254 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.411255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.412434 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.412436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413250 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.413326 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413339 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.413408 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.413462 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":89,"threads_started":1}
I20260928 06:33:37.413998 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39625 (local address 127.0.0.1:54558)
0928 06:33:37.409974 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.409985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.410019 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.410435 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.410436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.410438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:37.410482 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.410484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411734 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.411735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.412805 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.412808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413728 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.413803 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413806 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.413854 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.413903 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":6,"thread_start_us":89,"threads_started":1}
I20260928 06:33:37.414042 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.24.11.130:37613)
0928 06:33:37.409876 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.409882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.409892 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:37.409935 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.409938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.409957 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.410062 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:37.410434 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411572 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.412805 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413567 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413637 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413688 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.413904 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.413949 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260928 06:33:37.414484 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.24.11.129:39625)
0928 06:33:37.410205 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.410210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.410218 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:33:37.410252 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.410255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.410270 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.410354 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:37.410567 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411653 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.412883 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413574 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413636 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413689 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.414329 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.414393 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":74,"threads_started":1}
I20260928 06:33:37.413997 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.409389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.0.0.1:43782)
0928 06:33:37.409563 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.409606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.409674 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.410304 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.410304 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.410306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:37.410351 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.410352 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.411680 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.411681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.412717 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.412718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.413727 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.413728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.413798 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.413802 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.413861 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.413903 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":35,"thread_start_us":50,"threads_started":1}
I20260928 06:33:37.414711 6299 tablet_service.cc:1467] Processing CreateTablet for tablet 30053af1f61346de825dda841306bf28 (DEFAULT_TABLE table=test-table [id=d5b3e5d6b562415988ae08cb5894659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:37.415094 6299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30053af1f61346de825dda841306bf28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:37.415340 6031 tablet_service.cc:1467] Processing CreateTablet for tablet 30053af1f61346de825dda841306bf28 (DEFAULT_TABLE table=test-table [id=d5b3e5d6b562415988ae08cb5894659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:37.415341 6165 tablet_service.cc:1467] Processing CreateTablet for tablet 30053af1f61346de825dda841306bf28 (DEFAULT_TABLE table=test-table [id=d5b3e5d6b562415988ae08cb5894659f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:37.415648 6031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30053af1f61346de825dda841306bf28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:37.415649 6165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30053af1f61346de825dda841306bf28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:37.417870 6382 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Bootstrap starting.
I20260928 06:33:37.418462 6382 tablet_bootstrap.cc:654] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:37.418947 6384 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap starting.
I20260928 06:33:37.419042 6382 log.cc:824] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:37.419615 6384 tablet_bootstrap.cc:654] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:37.419930 6382 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: No bootstrap required, opened a new log
I20260928 06:33:37.420019 6382 ts_tablet_manager.cc:1397] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:37.420179 6384 log.cc:824] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:37.420533 6383 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Bootstrap starting.
I20260928 06:33:37.420886 6384 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: No bootstrap required, opened a new log
I20260928 06:33:37.421008 6384 ts_tablet_manager.cc:1397] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:37.421243 6383 tablet_bootstrap.cc:654] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:37.421869 6383 log.cc:824] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:37.422230 6382 raft_consensus.cc:348] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.422403 6382 raft_consensus.cc:374] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:37.422477 6382 raft_consensus.cc:729] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c874a4df04beab836e8db1a1997c0, State: Initialized, Role: FOLLOWER
I20260928 06:33:37.422605 6382 consensus_queue.cc:260] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.422650 6383 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: No bootstrap required, opened a new log
I20260928 06:33:37.422710 6383 ts_tablet_manager.cc:1397] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:33:37.422899 6382 ts_tablet_manager.cc:1428] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:37.423022 6365 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
I20260928 06:33:37.423012 6384 raft_consensus.cc:348] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.423171 6384 raft_consensus.cc:374] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:37.423239 6384 raft_consensus.cc:729] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39125b0dd79345ed9279d17c9f558ac8, State: Initialized, Role: FOLLOWER
I20260928 06:33:37.423394 6384 consensus_queue.cc:260] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.423691 6097 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
I20260928 06:33:37.423960 6384 ts_tablet_manager.cc:1428] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:37.424902 6383 raft_consensus.cc:348] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.425028 6383 raft_consensus.cc:374] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:37.425050 6383 raft_consensus.cc:729] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2878ceab4bb467cafddb7223a4e709c, State: Initialized, Role: FOLLOWER
I20260928 06:33:37.425172 6383 consensus_queue.cc:260] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.425462 6383 ts_tablet_manager.cc:1428] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:33:37.425477 6231 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
W20260928 06:33:37.467641 6232 tablet.cc:2369] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:33:37.551884 6098 tablet.cc:2369] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:37.593361 6388 raft_consensus.cc:482] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:37.593497 6388 raft_consensus.cc:504] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.594023 6388 leader_election.cc:288] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613), 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625)
I20260928 06:33:37.596786 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.594299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:34177 (local address 127.24.11.129:39625)
0928 06:33:37.594342 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.594344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.594354 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:37.594403 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.594404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.594416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.594491 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:37.594643 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.595126 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596033 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.596383 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596468 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.596476 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.596679 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.596728 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:33:37.596778 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.594211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39625 (local address 127.24.11.131:34177)
0928 06:33:37.594281 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.594294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.594337 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.594492 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.594493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.594495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:37.594572 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.594574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.595219 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.595220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.595941 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.595942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.596464 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.596545 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.596549 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.596656 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.596705 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:33:37.597038 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.594302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:57973 (local address 127.24.11.130:37613)
0928 06:33:37.594348 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.594351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.594480 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:33:37.594550 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.594551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.594565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.594636 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:37.594842 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.595309 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596230 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.596651 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.596749 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.596920 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.596970 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:33:37.597121 6051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30053af1f61346de825dda841306bf28" candidate_uuid: "db2c874a4df04beab836e8db1a1997c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39125b0dd79345ed9279d17c9f558ac8" is_pre_election: true
I20260928 06:33:37.597250 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.594231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.24.11.131:57973)
0928 06:33:37.594410 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.594418 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:37.594493 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:37.594638 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:37.594638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:37.594640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:37.594760 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.594762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.595394 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.595395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.596130 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:37.596131 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.596744 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.596745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:37.596820 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.596826 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:33:37.597132 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 06:33:37.597175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260928 06:33:37.597311 6051 raft_consensus.cc:2454] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db2c874a4df04beab836e8db1a1997c0 in term 0.
I20260928 06:33:37.597545 6185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30053af1f61346de825dda841306bf28" candidate_uuid: "db2c874a4df04beab836e8db1a1997c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2878ceab4bb467cafddb7223a4e709c" is_pre_election: true
I20260928 06:33:37.597710 6255 leader_election.cc:302] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8, db2c874a4df04beab836e8db1a1997c0; no voters:
I20260928 06:33:37.597730 6185 raft_consensus.cc:2454] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db2c874a4df04beab836e8db1a1997c0 in term 0.
I20260928 06:33:37.597849 6388 raft_consensus.cc:2781] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:37.597958 6388 raft_consensus.cc:482] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:37.598001 6388 raft_consensus.cc:3037] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:37.598716 6388 raft_consensus.cc:504] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.598866 6388 leader_election.cc:288] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [CANDIDATE]: Term 1 election: Requested vote from peers f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613), 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625)
I20260928 06:33:37.599066 6185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30053af1f61346de825dda841306bf28" candidate_uuid: "db2c874a4df04beab836e8db1a1997c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2878ceab4bb467cafddb7223a4e709c"
I20260928 06:33:37.599073 6051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30053af1f61346de825dda841306bf28" candidate_uuid: "db2c874a4df04beab836e8db1a1997c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39125b0dd79345ed9279d17c9f558ac8"
I20260928 06:33:37.599133 6185 raft_consensus.cc:3037] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:37.599156 6051 raft_consensus.cc:3037] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:37.599877 6185 raft_consensus.cc:2454] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db2c874a4df04beab836e8db1a1997c0 in term 1.
I20260928 06:33:37.599886 6051 raft_consensus.cc:2454] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db2c874a4df04beab836e8db1a1997c0 in term 1.
I20260928 06:33:37.600052 6254 leader_election.cc:302] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c; no voters:
I20260928 06:33:37.600153 6388 raft_consensus.cc:2781] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:37.600310 6388 raft_consensus.cc:686] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 1 LEADER]: Becoming Leader. State: Replica: db2c874a4df04beab836e8db1a1997c0, State: Running, Role: LEADER
I20260928 06:33:37.600423 6388 consensus_queue.cc:237] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:37.601276 5905 catalog_manager.cc:5495] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to db2c874a4df04beab836e8db1a1997c0 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "db2c874a4df04beab836e8db1a1997c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } health_report { overall_health: UNKNOWN } } }
W20260928 06:33:37.624739 6366 tablet.cc:2369] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:37.639859 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.637580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.24.11.131:37215)
0928 06:33:37.637643 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.637644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:37.637654 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:37.637701 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:37.637701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:37.637719 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:37.637780 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:37.637922 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.638387 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.639125 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:37.639303 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:37.639353 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:37.639526 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:37.639628 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:37.639689 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:37.639763 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:33:37.639780 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:33:37.641623 6185 raft_consensus.cc:1261] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Refusing update from remote peer db2c874a4df04beab836e8db1a1997c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:37.641702 6051 raft_consensus.cc:1261] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Refusing update from remote peer db2c874a4df04beab836e8db1a1997c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:37.641997 6392 consensus_queue.cc:1035] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:37.642156 6388 consensus_queue.cc:1035] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:37.644832 6397 mvcc.cc:205] Tried to move back new op lower bound from 7334204283457359872 to 7334204283292131328. Current Snapshot: MvccSnapshot[applied={T|T < 7334204283457359872}]
I20260928 06:33:37.646533 5905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:37.646633 5905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:37.647395 5905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Sending DeleteTablet for 3 replicas of tablet 30053af1f61346de825dda841306bf28
I20260928 06:33:37.647691 6299 tablet_service.cc:1514] Processing DeleteTablet for tablet 30053af1f61346de825dda841306bf28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:37 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:59646
I20260928 06:33:37.647780 6165 tablet_service.cc:1514] Processing DeleteTablet for tablet 30053af1f61346de825dda841306bf28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:37 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43782
I20260928 06:33:37.647799 6031 tablet_service.cc:1514] Processing DeleteTablet for tablet 30053af1f61346de825dda841306bf28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:37 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54558
I20260928 06:33:37.648012 6405 tablet_replica.cc:330] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: stopping tablet replica
I20260928 06:33:37.648068 6406 tablet_replica.cc:330] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: stopping tablet replica
I20260928 06:33:37.648070 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5968
I20260928 06:33:37.648789 6405 raft_consensus.cc:2229] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:33:37.648970 6405 raft_consensus.cc:2258] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:37.649681 6405 ts_tablet_manager.cc:1905] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:37.651710 6405 ts_tablet_manager.cc:1918] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:37.651788 6405 log.cc:1196] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/30053af1f61346de825dda841306bf28
I20260928 06:33:37.652071 6405 ts_tablet_manager.cc:1939] T 30053af1f61346de825dda841306bf28 P db2c874a4df04beab836e8db1a1997c0: Deleting consensus metadata
I20260928 06:33:37.652498 5891 catalog_manager.cc:4841] TS db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): tablet 30053af1f61346de825dda841306bf28 (table test-table [id=d5b3e5d6b562415988ae08cb5894659f]) successfully deleted
W20260928 06:33:37.653721 5893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:39625 (error 108)
I20260928 06:33:37.653821 6407 tablet_replica.cc:330] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: stopping tablet replica
I20260928 06:33:37.653950 6407 raft_consensus.cc:2229] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:37.654093 6407 raft_consensus.cc:2258] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:37.654312 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6102
I20260928 06:33:37.654418 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.654197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39625 (local address 127.0.0.1:54562)
0928 06:33:37.654320 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.654366 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 06:33:37.654899 5893 catalog_manager.cc:4584] TS 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30053af1f61346de825dda841306bf28: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39625: connect: Connection refused (error 111)
I20260928 06:33:37.660938 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.660803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.0.0.1:43794)
0928 06:33:37.660863 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:37.660899 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:33:37.660964 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6236
I20260928 06:33:37.666213 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 5874
I20260928 06:33:37.671607 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--webserver_interface=127.24.11.190
--webserver_port=40967
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:37.751682 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.751892 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.751915 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.753057 6409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:37.753100 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.753115 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.753127 6409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:37.753180 6409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:37.754645 6409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=40967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:37.754920 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.755164 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:37.767795 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.767913 6409 server_base.cc:1034] running on GCE node
W20260928 06:33:37.767913 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.767813 6418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.768368 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.768723 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.769892 6409 hybrid_clock.cc:648] HybridClock initialized: now 1790577217769880 us; error 48 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.776790 6409 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:37.777289 6409 webserver.cc:460] Webserver started at http://127.24.11.190:40967/ using document root <none> and password file <none>
I20260928 06:33:37.777612 6409 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.777697 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.779198 6409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:37.779994 6425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.780217 6409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:37.780354 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "28ea00026d4749a2be2bec16aacedc95"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-38pg"
server_key: "0210981929075a00b8f2af5c5a5c9925"
server_key_iv: "d9f9f8def0c6f5cb10c7e03ddee03c10"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.780833 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.816398 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.818909 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.819079 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.823451 6409 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:42751
I20260928 06:33:37.823493 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:42751 every 8 connection(s)
I20260928 06:33:37.823885 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:37.825460 6478 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:37.826181 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap starting.
I20260928 06:33:37.827023 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6409
I20260928 06:33:37.827306 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:39625
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=38781
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:37.837735 6478 log.cc:824] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:37.840471 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:37.840840 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap complete.
I20260928 06:33:37.844165 6478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:37.844501 6478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Initialized, Role: FOLLOWER
I20260928 06:33:37.844659 6478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:37.844722 6478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:37.844777 6478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:37.844825 6478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:33:37.846236 6478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:37.846372 6478 leader_election.cc:302] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28ea00026d4749a2be2bec16aacedc95; no voters:
I20260928 06:33:37.846529 6478 leader_election.cc:288] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:33:37.846616 6482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:33:37.846782 6478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:37.846815 6482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 LEADER]: Becoming Leader. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Running, Role: LEADER
I20260928 06:33:37.847071 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:37.847417 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:37.847429 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28ea00026d4749a2be2bec16aacedc95. Latest consensus state: current_term: 2 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:37.847606 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:37.847613 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:37.847847 6491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:37.848840 6491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:37.848964 6491 catalog_manager.cc:1259] Loaded cluster ID: 4db13cc4b61d4ab18ddcb8fa618db0f8
I20260928 06:33:37.849017 6491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:33:37.849237 6491 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:33:37.852003 6491 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:33:37.853960 6491 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:37.854161 6491 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Loaded TSK: 0
I20260928 06:33:37.854354 6491 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:33:37.927531 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:37.927759 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:37.927788 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:37.929077 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:37.929121 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:37.929152 6480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:37.930845 6480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:39625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=38781
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:37.931084 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:37.931356 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:37.937357 6506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.937358 6508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:37.937337 6505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:37.937799 6480 server_base.cc:1034] running on GCE node
I20260928 06:33:37.938068 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:37.938345 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:37.939527 6480 hybrid_clock.cc:648] HybridClock initialized: now 1790577217939499 us; error 42 us; skew 500 ppm
Sep 28 06:33:37 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:37.942613 6480 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:37.943166 6480 webserver.cc:460] Webserver started at http://127.24.11.129:38781/ using document root <none> and password file <none>
I20260928 06:33:37.943425 6480 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:37.943513 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:37.945263 6480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:33:37.946094 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:37.946301 6480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:37.946398 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "39125b0dd79345ed9279d17c9f558ac8"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "3bafddd48bcc53dc1d4126414970ccc2"
server_key_iv: "d332344ec4932f980b0ada3bb2037ff3"
server_key_version: "encryptionkey@0"
I20260928 06:33:37.946839 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:37.976481 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:37.979195 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:37.979382 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:37.979676 6480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:37.980196 6522 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:33:37.981787 6480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:33:37.981849 6480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:33:37.981884 6480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:33:37.982712 6480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:33:37.982755 6480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:37.982846 6522 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap starting.
I20260928 06:33:37.991928 6480 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39625
I20260928 06:33:37.992525 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:37.993878 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6480
I20260928 06:33:37.994052 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39625 every 8 connection(s)
I20260928 06:33:37.994199 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:37613
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=42281
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:37.998901 6522 log.cc:824] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.002578 6522 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:38.002976 6522 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap complete.
I20260928 06:33:38.004951 6522 ts_tablet_manager.cc:1397] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.011s
I20260928 06:33:38.006551 6522 raft_consensus.cc:348] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:38.006956 6522 raft_consensus.cc:729] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39125b0dd79345ed9279d17c9f558ac8, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.007155 6522 consensus_queue.cc:260] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:38.007472 6522 ts_tablet_manager.cc:1428] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:38.019677 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.994682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:33927)
0928 06:33:38.002723 (+ 8041us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.002780 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.003668 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.004366 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.004370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.004791 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.005104 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.005112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.006934 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.006935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.008209 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.008212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.009341 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.009342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.009466 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.010137 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.010147 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.014898 (+ 4751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.017407 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.017416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.017419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.017839 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.018171 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.018171 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.018172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.018307 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.018832 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.018862 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.019124 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.019251 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.019438 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7966,"thread_start_us":102,"threads_started":1}
I20260928 06:33:38.019722 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:37.994758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33927 (local address 127.24.11.190:42751)
0928 06:33:37.998016 (+ 3258us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:37.998028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.002930 (+ 4902us) server_negotiation.cc:407] Connection header received
0928 06:33:38.003939 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.003948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.004017 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.004139 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.005261 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.006799 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.008312 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.009106 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.009245 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.015060 (+ 5815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.015065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.015066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.015087 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.017308 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.017953 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.017957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.017957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.018064 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.018391 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.018392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.018392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.018718 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.018802 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.019368 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.019523 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2859,"thread_start_us":85,"threads_started":1}
I20260928 06:33:38.021415 6630 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:38.021566 6630 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:38.021767 6630 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:38.022603 6442 ts_manager.cc:194] Registered new tserver with Master: 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625)
I20260928 06:33:38.023908 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:33927
I20260928 06:33:38.025084 6630 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577217, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.129@KRBTEST.COM
I20260928 06:33:38.034505 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.023373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39625 (local address 127.0.0.1:54576)
0928 06:33:38.023560 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.023605 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.023679 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.024597 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.024602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.024938 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.024998 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.025001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.026222 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.026223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.026821 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.026823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.027455 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.027456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.027536 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.028081 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.028087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.030712 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.033077 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.033100 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.033102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.033392 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.033692 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.033693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.033694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.033757 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.034098 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.034104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.034159 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.034249 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.034391 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260928 06:33:38.034955 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.024246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.24.11.129:39625)
0928 06:33:38.024404 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.024408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.024418 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:38.024455 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.024457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.024471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.024538 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.025092 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.026129 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.026911 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.027303 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.027386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.030850 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.030859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.030859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.030867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.032983 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.033499 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.033502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.033503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.033616 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.033822 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.033823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.033823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.034003 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.034073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.034756 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.034863 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260928 06:33:38.037938 6564 tablet_service.cc:1514] Processing DeleteTablet for tablet 30053af1f61346de825dda841306bf28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:37 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54576
I20260928 06:33:38.038712 6639 tablet_replica.cc:330] stopping tablet replica
I20260928 06:33:38.038839 6639 raft_consensus.cc:2229] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:38.038909 6639 raft_consensus.cc:2258] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:38.039222 6639 ts_tablet_manager.cc:1905] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:38.041252 6639 ts_tablet_manager.cc:1918] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:38.041343 6639 log.cc:1196] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/30053af1f61346de825dda841306bf28
I20260928 06:33:38.041662 6639 ts_tablet_manager.cc:1939] T 30053af1f61346de825dda841306bf28 P 39125b0dd79345ed9279d17c9f558ac8: Deleting consensus metadata
I20260928 06:33:38.042264 6429 catalog_manager.cc:4841] TS 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625): tablet 30053af1f61346de825dda841306bf28 (table test-table [id=d5b3e5d6b562415988ae08cb5894659f]) successfully deleted
W20260928 06:33:38.095230 6634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.095472 6634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.095510 6634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.096748 6634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.096813 6634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.096861 6634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:38.098390 6634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:37613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=42281
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:38.098656 6634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.098901 6634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.104120 6647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.104185 6644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.104127 6645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.104328 6634 server_base.cc:1034] running on GCE node
I20260928 06:33:38.104549 6634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.104828 6634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.106031 6634 hybrid_clock.cc:648] HybridClock initialized: now 1790577218106014 us; error 26 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.108697 6634 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:38.109193 6634 webserver.cc:460] Webserver started at http://127.24.11.130:42281/ using document root <none> and password file <none>
I20260928 06:33:38.109431 6634 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.109517 6634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.110960 6634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.111620 6654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.111797 6634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:38.111901 6634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f2878ceab4bb467cafddb7223a4e709c"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "1a3b15a50e2c1abffdb6472fbfe8804b"
server_key_iv: "07c605c582751785cb16101a3812d170"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.112277 6634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.136087 6634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.138628 6634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.138814 6634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.139077 6634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.139530 6661 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:33:38.141278 6634 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:33:38.141364 6634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:33:38.141413 6634 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:33:38.142108 6634 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:33:38.142164 6634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:38.142232 6661 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Bootstrap starting.
I20260928 06:33:38.149135 6634 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37613
I20260928 06:33:38.149276 6768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37613 every 8 connection(s)
I20260928 06:33:38.149660 6634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:38.151194 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6634
I20260928 06:33:38.151507 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:37215
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=40287
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:38.153224 6661 log.cc:824] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.155974 6661 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:38.156296 6661 tablet_bootstrap.cc:492] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Bootstrap complete.
I20260928 06:33:38.157579 6661 ts_tablet_manager.cc:1397] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.011s
I20260928 06:33:38.159024 6661 raft_consensus.cc:348] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:38.159407 6661 raft_consensus.cc:729] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2878ceab4bb467cafddb7223a4e709c, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.159554 6661 consensus_queue.cc:260] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } }
I20260928 06:33:38.159847 6661 ts_tablet_manager.cc:1428] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:38.163376 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.150656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:57579 (local address 127.24.11.190:42751)
0928 06:33:38.150711 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.150713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.151026 (+ 313us) server_negotiation.cc:407] Connection header received
0928 06:33:38.153772 (+ 2746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.153774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.153788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.153878 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.154740 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.155498 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.157142 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.157510 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.157586 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.160457 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.160459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.160460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.160468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.161951 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.162376 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.162377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.162377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.162442 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.162707 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.162708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.162708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.162827 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.162903 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.163181 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.163274 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:33:38.163655 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.150636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:57579)
0928 06:33:38.150827 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.150909 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.153633 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.153971 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.153976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.154326 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.154595 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.154602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.155643 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.155644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.157037 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.157040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.157702 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.157703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.157849 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.158541 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.158551 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.160353 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.162037 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.162044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.162047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.162273 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.162511 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.162512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.162512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.162628 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.162906 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.162914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.163050 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.163159 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.163314 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260928 06:33:38.164151 6769 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:38.164245 6769 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:38.164423 6769 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:38.164978 6442 ts_manager.cc:194] Registered new tserver with Master: f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
I20260928 06:33:38.165592 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:57579
I20260928 06:33:38.166303 6769 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
I20260928 06:33:38.172477 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.165262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.0.0.1:43796)
0928 06:33:38.165310 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.165319 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.165352 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.169828 (+ 4476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.169829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.169831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:38.169882 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.169884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.170755 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.170756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.171532 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.171534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.172177 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.172178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.172255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.172287 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.172346 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.172398 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:33:38.172478 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.165336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.24.11.130:37613)
0928 06:33:38.169660 (+ 4324us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.169664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.169674 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:33:38.169703 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.169704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.169717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.169815 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:38.170000 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.170656 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.171631 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.172094 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.172179 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.172238 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.172364 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.172414 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4287,"thread_start_us":67,"threads_started":1}
I20260928 06:33:38.172919 6703 tablet_service.cc:1514] Processing DeleteTablet for tablet 30053af1f61346de825dda841306bf28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:37 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43796
I20260928 06:33:38.173645 6776 tablet_replica.cc:330] stopping tablet replica
I20260928 06:33:38.173770 6776 raft_consensus.cc:2229] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:38.173862 6776 raft_consensus.cc:2258] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:38.174154 6776 ts_tablet_manager.cc:1905] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:38.176327 6776 ts_tablet_manager.cc:1918] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:38.176427 6776 log.cc:1196] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/30053af1f61346de825dda841306bf28
I20260928 06:33:38.176644 6776 ts_tablet_manager.cc:1939] T 30053af1f61346de825dda841306bf28 P f2878ceab4bb467cafddb7223a4e709c: Deleting consensus metadata
I20260928 06:33:38.177134 6428 catalog_manager.cc:4841] TS f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): tablet 30053af1f61346de825dda841306bf28 (table test-table [id=d5b3e5d6b562415988ae08cb5894659f]) successfully deleted
W20260928 06:33:38.238806 6772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.239034 6772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.239059 6772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.240278 6772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.240329 6772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.240387 6772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:38.241962 6772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:37215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=40287
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:38.242218 6772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.242461 6772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.247627 6782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.247592 6781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.247900 6772 server_base.cc:1034] running on GCE node
W20260928 06:33:38.247599 6784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.248245 6772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.248499 6772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.249696 6772 hybrid_clock.cc:648] HybridClock initialized: now 1790577218249692 us; error 38 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.252323 6772 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:38.252770 6772 webserver.cc:460] Webserver started at http://127.24.11.131:40287/ using document root <none> and password file <none>
I20260928 06:33:38.253002 6772 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.253084 6772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.254544 6772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.255138 6791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.255306 6772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:38.255411 6772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "db2c874a4df04beab836e8db1a1997c0"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "19a66e16dac034efa692dea8fb21d26b"
server_key_iv: "44fb4d9a1436249e8a0499c58aec3425"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.255762 6772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.280224 6772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.282279 6772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.282444 6772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.282683 6772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.283033 6772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:38.283089 6772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.283133 6772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:38.283160 6772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.289211 6772 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37215
I20260928 06:33:38.289258 6904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37215 every 8 connection(s)
I20260928 06:33:38.289624 6772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:38.297103 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6772
I20260928 06:33:38.298731 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.290389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:58391 (local address 127.24.11.190:42751)
0928 06:33:38.290460 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.290462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.290657 (+ 195us) server_negotiation.cc:407] Connection header received
0928 06:33:38.291136 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.291137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.291146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.291202 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.292110 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.292599 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.293608 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.293788 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.293840 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.295878 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.295879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.295879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.295886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.297175 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.297636 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.297638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.297638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.297735 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.298041 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.298042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.298042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.298188 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.298266 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.298527 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.298614 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:38.298838 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.290383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:58391)
0928 06:33:38.290532 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.290568 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.291047 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.291254 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.291257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.291836 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:38.292031 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.292036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.292677 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.292678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.293485 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.293487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.293864 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.293865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.293964 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.294429 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:38.294437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:38.295795 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:38.297266 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.297275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.297276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.297568 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.297811 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:38.297812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:38.297813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:38.297971 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:38.298260 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:38.298265 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:38.298392 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.298477 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.298601 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":86,"threads_started":1}
I20260928 06:33:38.299333 6905 heartbeater.cc:347] Connected to a master server at 127.24.11.190:42751
I20260928 06:33:38.299432 6905 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:38.299612 6905 heartbeater.cc:511] Master 127.24.11.190:42751 requested a full tablet report, sending...
I20260928 06:33:38.300005 6442 ts_manager.cc:194] Registered new tserver with Master: db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215)
I20260928 06:33:38.300483 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:58391
I20260928 06:33:38.304571 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.297572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.24.11.190:42751)
0928 06:33:38.297697 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.297699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.297791 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:33:38.297869 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.297870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.297882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.297960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:38.298703 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.299687 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.300550 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.300802 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.300882 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.302182 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:38.302184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:38.302184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:38.302191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:38.303409 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.303762 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.303763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.303764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.303821 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:38.304033 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:38.304034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:38.304034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:38.304146 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:38.304348 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.304387 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.304466 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 06:33:38.305291 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:38.308117 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.305639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.24.11.190:42751)
0928 06:33:38.305671 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.305672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.305844 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:33:38.305918 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.305918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.305952 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.306036 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:38.306134 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.306594 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.307337 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.307544 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.307628 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.307799 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:38.307902 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:38.307988 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.308026 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.308045 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:33:38.308393 6442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:38.310071 6442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:38.312211 6703 tablet_service.cc:1467] Processing CreateTablet for tablet 80176ac3b1204fafa116ec7f9085710b (DEFAULT_TABLE table=test-table [id=623bbd19776545e5b29b3092bff25d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:38.312258 6564 tablet_service.cc:1467] Processing CreateTablet for tablet 80176ac3b1204fafa116ec7f9085710b (DEFAULT_TABLE table=test-table [id=623bbd19776545e5b29b3092bff25d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:38.312415 6564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80176ac3b1204fafa116ec7f9085710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:38.312420 6703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80176ac3b1204fafa116ec7f9085710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:38.314049 6661 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Bootstrap starting.
I20260928 06:33:38.314127 6522 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap starting.
I20260928 06:33:38.314355 6661 tablet_bootstrap.cc:654] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:38.314393 6522 tablet_bootstrap.cc:654] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:38.315357 6522 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: No bootstrap required, opened a new log
I20260928 06:33:38.315426 6661 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: No bootstrap required, opened a new log
I20260928 06:33:38.315433 6522 ts_tablet_manager.cc:1397] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.315469 6661 ts_tablet_manager.cc:1397] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:38.315579 6661 raft_consensus.cc:348] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.315582 6522 raft_consensus.cc:348] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.315644 6522 raft_consensus.cc:374] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:38.315644 6661 raft_consensus.cc:374] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:38.315689 6661 raft_consensus.cc:729] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2878ceab4bb467cafddb7223a4e709c, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.315689 6522 raft_consensus.cc:729] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39125b0dd79345ed9279d17c9f558ac8, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.315730 6661 consensus_queue.cc:260] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.315734 6522 consensus_queue.cc:260] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.315800 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.311980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:37215 (local address 127.0.0.1:59670)
0928 06:33:38.312036 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.312048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.312089 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.312473 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.312474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.312476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:38.312515 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.312517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.313455 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.313456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.314475 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.314478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.315550 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.315552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.315621 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.315625 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.315683 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.315733 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:33:38.315821 6522 ts_tablet_manager.cc:1428] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.315822 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.312104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.24.11.131:37215)
0928 06:33:38.312284 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.312289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.312300 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:38.312340 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.312343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.312359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.312468 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:38.312573 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.313349 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.314582 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.315458 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.315551 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.315615 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.315698 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.315740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260928 06:33:38.315819 6661 ts_tablet_manager.cc:1428] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.316651 6839 tablet_service.cc:1467] Processing CreateTablet for tablet 80176ac3b1204fafa116ec7f9085710b (DEFAULT_TABLE table=test-table [id=623bbd19776545e5b29b3092bff25d5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:38.317050 6839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80176ac3b1204fafa116ec7f9085710b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:38.319837 6914 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Bootstrap starting.
I20260928 06:33:38.320474 6914 tablet_bootstrap.cc:654] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:38.320931 6914 log.cc:824] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.321990 6914 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: No bootstrap required, opened a new log
I20260928 06:33:38.322065 6914 ts_tablet_manager.cc:1397] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:38.323446 6914 raft_consensus.cc:348] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.323583 6914 raft_consensus.cc:374] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:38.323606 6914 raft_consensus.cc:729] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2c874a4df04beab836e8db1a1997c0, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.323755 6914 consensus_queue.cc:260] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.324038 6914 ts_tablet_manager.cc:1428] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:38.324077 6905 heartbeater.cc:503] Master 127.24.11.190:42751 was elected leader, sending a full tablet report...
W20260928 06:33:38.400542 6770 tablet.cc:2369] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:33:38.493367 6631 tablet.cc:2369] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:38.510639 6636 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:38.510780 6636 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.511094 6636 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
I20260928 06:33:38.515204 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.511388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:51877 (local address 127.24.11.130:37613)
0928 06:33:38.511459 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.511462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.511544 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:33:38.511633 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.511635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.511644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.511710 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:38.511900 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.512645 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514164 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.514761 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514835 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.514845 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.515084 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.515130 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:33:38.515208 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.511394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:57059 (local address 127.24.11.131:37215)
0928 06:33:38.511459 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.511462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.511547 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:33:38.511633 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.511634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.511644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.511710 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:38.511900 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.512637 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514165 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.514754 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514922 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.514929 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.515111 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.515150 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:33:38.515244 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.511293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.24.11.129:51877)
0928 06:33:38.511458 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.511470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.511553 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.511767 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.511767 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.511768 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:38.511819 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.511821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.512723 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.512724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.514066 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.514070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.514858 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.514970 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.515000 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.515068 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.515118 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":41}
I20260928 06:33:38.515267 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.511336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:37215 (local address 127.24.11.129:57059)
0928 06:33:38.511476 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.511485 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.511553 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.511732 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:38.511733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:38.511735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:38.511821 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.511822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.512723 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.512724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.514066 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:38.514070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.514946 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.514947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:38.515040 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.515044 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:38.515094 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:33:38.515121 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":60,"thread_start_us":71,"threads_started":1}
I20260928 06:33:38.515722 6723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80176ac3b1204fafa116ec7f9085710b" candidate_uuid: "39125b0dd79345ed9279d17c9f558ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2878ceab4bb467cafddb7223a4e709c" is_pre_election: true
I20260928 06:33:38.515733 6859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80176ac3b1204fafa116ec7f9085710b" candidate_uuid: "39125b0dd79345ed9279d17c9f558ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c874a4df04beab836e8db1a1997c0" is_pre_election: true
I20260928 06:33:38.515908 6859 raft_consensus.cc:2454] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39125b0dd79345ed9279d17c9f558ac8 in term 0.
I20260928 06:33:38.515914 6723 raft_consensus.cc:2454] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39125b0dd79345ed9279d17c9f558ac8 in term 0.
I20260928 06:33:38.516168 6518 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8, f2878ceab4bb467cafddb7223a4e709c; no voters:
I20260928 06:33:38.516325 6636 raft_consensus.cc:2781] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:38.516414 6636 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:38.516449 6636 raft_consensus.cc:3037] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:38.517187 6636 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.517351 6636 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 1 election: Requested vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
I20260928 06:33:38.517598 6859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80176ac3b1204fafa116ec7f9085710b" candidate_uuid: "39125b0dd79345ed9279d17c9f558ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2c874a4df04beab836e8db1a1997c0"
I20260928 06:33:38.517616 6723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80176ac3b1204fafa116ec7f9085710b" candidate_uuid: "39125b0dd79345ed9279d17c9f558ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2878ceab4bb467cafddb7223a4e709c"
I20260928 06:33:38.517694 6723 raft_consensus.cc:3037] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:38.517694 6859 raft_consensus.cc:3037] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:38.518566 6723 raft_consensus.cc:2454] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39125b0dd79345ed9279d17c9f558ac8 in term 1.
I20260928 06:33:38.518570 6859 raft_consensus.cc:2454] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39125b0dd79345ed9279d17c9f558ac8 in term 1.
I20260928 06:33:38.518777 6517 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8, db2c874a4df04beab836e8db1a1997c0; no voters:
I20260928 06:33:38.518903 6636 raft_consensus.cc:2781] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:38.519047 6636 raft_consensus.cc:686] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 LEADER]: Becoming Leader. State: Replica: 39125b0dd79345ed9279d17c9f558ac8, State: Running, Role: LEADER
I20260928 06:33:38.519130 6636 consensus_queue.cc:237] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.519977 6441 catalog_manager.cc:5495] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129). New cstate: current_term: 1 leader_uuid: "39125b0dd79345ed9279d17c9f558ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } health_report { overall_health: UNKNOWN } } }
W20260928 06:33:38.540153 6906 tablet.cc:2369] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:38.544802 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.542278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.24.11.129:39625)
0928 06:33:38.542426 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:38.542433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:38.542442 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:38.542478 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:38.542478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:38.542497 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:38.542581 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:38.542760 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.543236 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.543984 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:38.544189 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:38.544268 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:38.544452 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:38.544554 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:38.544641 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:38.544681 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:33:38.544703 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 06:33:38.546365 6859 raft_consensus.cc:1261] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Refusing update from remote peer 39125b0dd79345ed9279d17c9f558ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:38.546365 6723 raft_consensus.cc:1261] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Refusing update from remote peer 39125b0dd79345ed9279d17c9f558ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:38.546679 6636 consensus_queue.cc:1035] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:38.546782 6918 consensus_queue.cc:1035] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:38.549119 6922 mvcc.cc:205] Tried to move back new op lower bound from 7334204287163355136 to 7334204287055171584. Current Snapshot: MvccSnapshot[applied={T|T < 7334204287163355136}]
I20260928 06:33:38.549638 6923 mvcc.cc:205] Tried to move back new op lower bound from 7334204287163355136 to 7334204287055171584. Current Snapshot: MvccSnapshot[applied={T|T < 7334204287163355136}]
I20260928 06:33:38.552052 6442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:38.552131 6442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:38.552909 6442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Sending DeleteTablet for 3 replicas of tablet 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:38.553216 6839 tablet_service.cc:1514] Processing DeleteTablet for tablet 80176ac3b1204fafa116ec7f9085710b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:38 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 06:33:38.553242 6703 tablet_service.cc:1514] Processing DeleteTablet for tablet 80176ac3b1204fafa116ec7f9085710b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:38 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43796
I20260928 06:33:38.553298 6564 tablet_service.cc:1514] Processing DeleteTablet for tablet 80176ac3b1204fafa116ec7f9085710b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:38 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:54576
I20260928 06:33:38.553354 6776 tablet_replica.cc:330] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: stopping tablet replica
I20260928 06:33:38.553424 6776 raft_consensus.cc:2229] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:38.553462 6931 tablet_replica.cc:330] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: stopping tablet replica
I20260928 06:33:38.553483 6776 raft_consensus.cc:2258] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:38.553507 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6480
I20260928 06:33:38.553578 6931 raft_consensus.cc:2229] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:38.553682 6931 raft_consensus.cc:2258] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:38.554550 6776 ts_tablet_manager.cc:1905] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:38.555109 6931 ts_tablet_manager.cc:1905] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:33:38.556353 6776 ts_tablet_manager.cc:1918] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:38.556427 6776 log.cc:1196] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:38.556644 6776 ts_tablet_manager.cc:1939] T 80176ac3b1204fafa116ec7f9085710b P f2878ceab4bb467cafddb7223a4e709c: Deleting consensus metadata
I20260928 06:33:38.557093 6428 catalog_manager.cc:4841] TS f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): tablet 80176ac3b1204fafa116ec7f9085710b (table test-table [id=623bbd19776545e5b29b3092bff25d5a]) successfully deleted
I20260928 06:33:38.557379 6931 ts_tablet_manager.cc:1918] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:38.557461 6931 log.cc:1196] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:38.557817 6931 ts_tablet_manager.cc:1939] T 80176ac3b1204fafa116ec7f9085710b P db2c874a4df04beab836e8db1a1997c0: Deleting consensus metadata
I20260928 06:33:38.558261 6427 catalog_manager.cc:4841] TS db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): tablet 80176ac3b1204fafa116ec7f9085710b (table test-table [id=623bbd19776545e5b29b3092bff25d5a]) successfully deleted
W20260928 06:33:38.560299 6429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:39625 (error 108)
I20260928 06:33:38.560554 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6634
I20260928 06:33:38.561213 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.560667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39625 (local address 127.0.0.1:54598)
0928 06:33:38.561141 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.561185 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455}
W20260928 06:33:38.561342 6429 catalog_manager.cc:4584] TS 39125b0dd79345ed9279d17c9f558ac8 (127.24.11.129:39625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80176ac3b1204fafa116ec7f9085710b: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39625: connect: Connection refused (error 111)
I20260928 06:33:38.565992 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6772
I20260928 06:33:38.571244 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 6409
I20260928 06:33:38.576547 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--webserver_interface=127.24.11.190
--webserver_port=40967
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:38.656308 6934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.656520 6934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.656543 6934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.658311 6934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:38.658365 6934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.658378 6934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.658391 6934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:38.658404 6934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:38.659967 6934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=40967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:38.660228 6934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.660481 6934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.665419 6940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.665442 6942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.665678 6934 server_base.cc:1034] running on GCE node
W20260928 06:33:38.665441 6939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.666038 6934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.666322 6934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.667488 6934 hybrid_clock.cc:648] HybridClock initialized: now 1790577218667476 us; error 30 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.670154 6934 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:38.670608 6934 webserver.cc:460] Webserver started at http://127.24.11.190:40967/ using document root <none> and password file <none>
I20260928 06:33:38.670887 6934 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.670962 6934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.672289 6934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:33:38.673031 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.673285 6934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.673404 6934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "28ea00026d4749a2be2bec16aacedc95"
format_stamp: "Formatted at 2026-09-28 06:33:36 on dist-test-slave-38pg"
server_key: "0210981929075a00b8f2af5c5a5c9925"
server_key_iv: "d9f9f8def0c6f5cb10c7e03ddee03c10"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.673828 6934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.682677 6934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.684465 6934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.684624 6934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.688685 6934 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:42751
I20260928 06:33:38.688719 7002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:42751 every 8 connection(s)
I20260928 06:33:38.689092 6934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:38.690639 7003 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:33:38.691409 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 6934
I20260928 06:33:38.691409 7003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap starting.
I20260928 06:33:38.691699 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:39625
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=38781
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:38.700860 7003 log.cc:824] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.703142 7003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:38.703462 7003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95: Bootstrap complete.
I20260928 06:33:38.705993 7003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:38.706210 7003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.706367 7003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:38.706439 7003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:38.706501 7003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:38.706542 7003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:33:38.707522 7003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:38.707608 7003 leader_election.cc:302] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28ea00026d4749a2be2bec16aacedc95; no voters:
I20260928 06:33:38.707789 7003 leader_election.cc:288] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:33:38.707828 7007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:33:38.707937 7007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [term 3 LEADER]: Becoming Leader. State: Replica: 28ea00026d4749a2be2bec16aacedc95, State: Running, Role: LEADER
I20260928 06:33:38.708020 7007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } }
I20260928 06:33:38.708076 7003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:38.708279 7009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28ea00026d4749a2be2bec16aacedc95. Latest consensus state: current_term: 3 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:38.708371 7009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:38.708285 7008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "28ea00026d4749a2be2bec16aacedc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28ea00026d4749a2be2bec16aacedc95" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 42751 } } }
I20260928 06:33:38.708544 7008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28ea00026d4749a2be2bec16aacedc95 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:38.708681 7011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:38.709473 7011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:38.709656 7011 catalog_manager.cc:1259] Loaded cluster ID: 4db13cc4b61d4ab18ddcb8fa618db0f8
I20260928 06:33:38.709903 7011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:33:38.710187 7011 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2c092379ad google::LogMessage::Fail() at ??:0
@ 0x7f2c0923a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f2c092374de google::LogMessage::Flush() at ??:0
@ 0x7f2c09238a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2c111580f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2c111618b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2c0b1050d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2c0b0fa954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2c0bf5e6db start_thread at ??:0
@ 0x7f2c07d3561f clone at ??:0
W20260928 06:33:38.788708 7005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.788961 7005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.789002 7005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.790251 7005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.790319 7005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.790365 7005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:38.791886 7005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:39625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=38781
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:38.792155 7005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.792397 7005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.797523 7040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.797654 7041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.797740 7005 server_base.cc:1034] running on GCE node
W20260928 06:33:38.797515 7043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.798028 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.798316 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.799503 7005 hybrid_clock.cc:648] HybridClock initialized: now 1790577218799488 us; error 28 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.802824 7005 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:38.803401 7005 webserver.cc:460] Webserver started at http://127.24.11.129:38781/ using document root <none> and password file <none>
I20260928 06:33:38.803674 7005 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.803742 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.805722 7005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:33:38.806586 7050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.806769 7005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.806862 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "39125b0dd79345ed9279d17c9f558ac8"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "3bafddd48bcc53dc1d4126414970ccc2"
server_key_iv: "d332344ec4932f980b0ada3bb2037ff3"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.807194 7005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.826319 7005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.828732 7005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.828871 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.829131 7005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.829607 7057 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:33:38.831664 7005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:33:38.831728 7005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:33:38.831771 7005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:33:38.832407 7005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:33:38.832463 7005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:33:38.832520 7057 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap starting.
I20260928 06:33:38.839629 7005 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39625
I20260928 06:33:38.839783 7164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39625 every 8 connection(s)
I20260928 06:33:38.840049 7005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:38.844015 7057 log.cc:824] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:38.846683 7057 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:33:38.847057 7057 tablet_bootstrap.cc:492] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Bootstrap complete.
I20260928 06:33:38.847143 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7005
I20260928 06:33:38.847383 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:37613
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=42281
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:33:38.848351 7057 ts_tablet_manager.cc:1397] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20260928 06:33:38.849900 7057 raft_consensus.cc:348] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.850221 7057 raft_consensus.cc:729] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39125b0dd79345ed9279d17c9f558ac8, State: Initialized, Role: FOLLOWER
I20260928 06:33:38.850385 7057 consensus_queue.cc:260] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:38.850660 7057 ts_tablet_manager.cc:1428] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260928 06:33:38.903852 7167 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:33:38.840683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:45631)
0928 06:33:38.840847 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.840890 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:38.841347 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:38.903654 (+ 62307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
W20260928 06:33:38.904100 7165 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:42751 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:42751: Client connection negotiation failed: client connection to 127.24.11.190:42751: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:33:38.904476 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.904364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:41535)
0928 06:33:38.904416 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.904457 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:33:38.904810 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.904729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:43269)
0928 06:33:38.904773 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.904795 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:33:38.904938 7165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:33:38.931537 7169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:38.931779 7169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:38.931819 7169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:38.933010 7169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:38.933077 7169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:38.933122 7169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:38.934651 7169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:37613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=42281
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:38.934924 7169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:38.935158 7169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:38.940166 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.940243 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:38.940326 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:38.940376 7169 server_base.cc:1034] running on GCE node
I20260928 06:33:38.940593 7169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:38.940868 7169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:38.942009 7169 hybrid_clock.cc:648] HybridClock initialized: now 1790577218941999 us; error 27 us; skew 500 ppm
Sep 28 06:33:38 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:38.944442 7169 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:38.944900 7169 webserver.cc:460] Webserver started at http://127.24.11.130:42281/ using document root <none> and password file <none>
I20260928 06:33:38.945127 7169 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:38.945194 7169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:38.946554 7169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:38.947156 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.947320 7169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:38.947425 7169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f2878ceab4bb467cafddb7223a4e709c"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "1a3b15a50e2c1abffdb6472fbfe8804b"
server_key_iv: "07c605c582751785cb16101a3812d170"
server_key_version: "encryptionkey@0"
I20260928 06:33:38.947768 7169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:38.975710 7169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:38.977624 7169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:38.977794 7169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:38.978037 7169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:38.978343 7169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:38.978397 7169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.978439 7169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:38.978482 7169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:38.984390 7169 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:37613
I20260928 06:33:38.984440 7298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:37613 every 8 connection(s)
I20260928 06:33:38.984766 7169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:38.985818 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.985414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:35577)
0928 06:33:38.985605 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.985682 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
W20260928 06:33:38.986233 7299 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:42751 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:42751: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
I20260928 06:33:38.986464 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.986396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:36833)
0928 06:33:38.986423 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.986447 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 06:33:38.986696 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:38.986649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:53715)
0928 06:33:38.986668 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:38.986685 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 06:33:38.986795 7299 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:33:38.993633 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7169
I20260928 06:33:38.993860 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:37215
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:42751
--webserver_port=40287
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:39419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:39.076164 7302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:39.076393 7302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:39.076418 7302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:39.077522 7302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:39.077618 7302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:39.077663 7302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:39.079092 7302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:39419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:37215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=40287
--tserver_master_addrs=127.24.11.190:42751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:39.079303 7302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:39.079486 7302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:33:39.084511 7302 server_base.cc:1034] running on GCE node
W20260928 06:33:39.084442 7307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.084450 7308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:39.084441 7310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:39.084865 7302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:39.085122 7302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:39.086298 7302 hybrid_clock.cc:648] HybridClock initialized: now 1790577219086272 us; error 38 us; skew 500 ppm
Sep 28 06:33:39 dist-test-slave-38pg krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:39.088861 7302 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:39.089334 7302 webserver.cc:460] Webserver started at http://127.24.11.131:40287/ using document root <none> and password file <none>
I20260928 06:33:39.089596 7302 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:39.089648 7302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:39.090934 7302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:39.091491 7317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.091673 7302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:39.091758 7302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "db2c874a4df04beab836e8db1a1997c0"
format_stamp: "Formatted at 2026-09-28 06:33:37 on dist-test-slave-38pg"
server_key: "19a66e16dac034efa692dea8fb21d26b"
server_key_iv: "44fb4d9a1436249e8a0499c58aec3425"
server_key_version: "encryptionkey@0"
I20260928 06:33:39.092087 7302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:39.098927 7302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:39.101137 7302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:39.101244 7302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:39.101424 7302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:39.101758 7302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:39.101792 7302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.101846 7302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:39.101871 7302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:39.107653 7302 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:37215
I20260928 06:33:39.107761 7430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:37215 every 8 connection(s)
I20260928 06:33:39.108081 7302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:39.108422 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7302
I20260928 06:33:39.109238 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.108821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:40985)
0928 06:33:39.109003 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.109096 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260928 06:33:39.109691 7431 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:42751 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:42751: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
I20260928 06:33:39.110029 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.109928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:53009)
0928 06:33:39.109986 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.110012 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:33:39.110319 7434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.110245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:56641)
0928 06:33:39.110285 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.110305 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:33:39.110438 7431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:33:39.905659 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.905337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:35685)
0928 06:33:39.905488 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.905588 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260928 06:33:39.987377 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:39.987143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:56391)
0928 06:33:39.987283 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:39.987324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260928 06:33:40.110965 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.110763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:43757)
0928 06:33:40.110876 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.110915 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260928 06:33:40.224967 7438 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:40.225090 7438 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:40.225423 7438 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
Sep 28 06:33:40 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.130@KRBTEST.COM
Sep 28 06:33:40 dist-test-slave-38pg krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577218, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.131@KRBTEST.COM
I20260928 06:33:40.239205 7440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.225741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:35709 (local address 127.24.11.131:37215)
0928 06:33:40.225936 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.225949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.225986 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:33:40.226575 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.226578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.226638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.226799 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.227910 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.229347 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.230975 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.231597 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.231728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.235379 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.235384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.235384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:40.235397 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.237272 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:40.237837 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:40.237841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:40.237841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:40.237955 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:40.238222 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:40.238223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:40.238223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:40.238533 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.238652 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.238974 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.239101 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
I20260928 06:33:40.239194 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.225655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:37215 (local address 127.24.11.129:35709)
0928 06:33:40.225719 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.225732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.225797 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.226877 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.226880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.227452 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:40.227734 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.227743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.229477 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.229478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.230858 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.230862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.231712 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.231714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.231858 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.232738 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:40.232744 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:40.235265 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:40.237340 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:40.237341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:40.237342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:40.237748 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:40.238030 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:40.238031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:40.238031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:40.238147 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:40.238633 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:40.238639 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:40.238795 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:33:40.238893 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:33:40.239080 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":85}
I20260928 06:33:40.239367 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.225810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:54521 (local address 127.24.11.130:37613)
0928 06:33:40.225936 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:40.225949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:40.225986 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:33:40.226575 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:40.226578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:40.226638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:40.226799 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:40.227910 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.229347 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.231042 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.231720 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.231852 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.235031 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:40.235036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:40.235036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:40.235063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:40.237191 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:40.237867 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:40.237871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:40.237871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:40.238006 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:40.238227 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:40.238228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:40.238228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:40.238589 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:40.238675 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:40.239042 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 06:33:40.239201 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260928 06:33:40.239521 7439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.225667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:37613 (local address 127.24.11.129:54521)
0928 06:33:40.225900 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.225911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:40.225971 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:40.226877 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:40.226880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:40.227452 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:40.227734 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.227743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.229486 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.229486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.230858 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:40.230862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:40.231944 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:40.231945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:40.232008 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:40.232544 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:40.232555 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:40.234859 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:40.237311 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:40.237319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:40.237321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:40.237753 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:40.238085 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:40.238086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:40.238086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:40.238152 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:40.238709 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:40.238710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:40.238799 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 06:33:40.238887 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:33:40.239383 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":424,"thread_start_us":69,"threads_started":1}
W20260928 06:33:40.240835 7053 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
W20260928 06:33:40.240967 7052 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:40.241017 7052 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8; no voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c
I20260928 06:33:40.241149 7438 raft_consensus.cc:2726] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:40.906430 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.906211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:40857)
0928 06:33:40.906343 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.906384 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 06:33:40.988103 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:40.987834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:48125)
0928 06:33:40.987974 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:40.988047 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 06:33:41.111564 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:41.111372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:56837)
0928 06:33:41.111486 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:41.111523 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260928 06:33:41.907213 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:41.906999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:38691)
0928 06:33:41.907124 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:41.907167 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260928 06:33:41.988857 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:41.988631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:54249)
0928 06:33:41.988783 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:41.988815 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260928 06:33:42.112284 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.112086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:32915)
0928 06:33:42.112202 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:42.112245 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260928 06:33:42.116459 7448 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:42.116560 7448 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:42.116755 7448 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
W20260928 06:33:42.117200 7052 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
W20260928 06:33:42.117264 7053 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:42.117303 7053 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8; no voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c
I20260928 06:33:42.117386 7448 raft_consensus.cc:2726] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:42.908138 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.907837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:55719)
0928 06:33:42.908028 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:42.908078 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260928 06:33:42.989782 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:42.989517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:35651)
0928 06:33:42.989699 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:42.989741 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260928 06:33:43.113004 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.112754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:42931)
0928 06:33:43.112924 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:43.112965 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260928 06:33:43.908962 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.908721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:56439)
0928 06:33:43.908867 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:43.908913 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260928 06:33:43.990515 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:43.990265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:34443)
0928 06:33:43.990429 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:43.990468 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260928 06:33:44.113693 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.113437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:57179)
0928 06:33:44.113613 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:44.113648 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260928 06:33:44.178668 7455 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:44.178767 7455 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:44.178946 7455 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
W20260928 06:33:44.179344 7053 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
W20260928 06:33:44.179440 7052 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:44.179505 7052 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8; no voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c
I20260928 06:33:44.179590 7455 raft_consensus.cc:2726] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:44.909802 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.909461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:43999)
0928 06:33:44.909658 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:44.909703 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260928 06:33:44.991206 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:44.990986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:57373)
0928 06:33:44.991125 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:44.991162 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 06:33:45.114466 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.114232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:54711)
0928 06:33:45.114377 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.114421 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 06:33:45.894410 7459 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:45.894508 7459 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:45.894774 7459 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
W20260928 06:33:45.895058 7052 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
W20260928 06:33:45.895154 7053 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:45.895188 7053 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8; no voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c
I20260928 06:33:45.895255 7459 raft_consensus.cc:2726] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:45.910584 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.910277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:57333)
0928 06:33:45.910494 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.910540 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260928 06:33:45.991994 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:45.991713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:56773)
0928 06:33:45.991902 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:45.991950 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":64,"threads_started":1}
I20260928 06:33:46.115173 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.114958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:51867)
0928 06:33:46.115078 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.115120 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260928 06:33:46.911334 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.911096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:41471)
0928 06:33:46.911241 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.911287 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260928 06:33:46.992717 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:46.992477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:55245)
0928 06:33:46.992627 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:46.992670 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260928 06:33:47.115869 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.115659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:54919)
0928 06:33:47.115783 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.115822 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260928 06:33:47.912261 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.911965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:44369)
0928 06:33:47.912147 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.912203 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260928 06:33:47.993526 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:47.993283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:41735)
0928 06:33:47.993437 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:47.993480 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260928 06:33:48.116485 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.116290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:59029)
0928 06:33:48.116407 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.116443 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 06:33:48.912998 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.912759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:39185)
0928 06:33:48.912907 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.912952 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260928 06:33:48.994266 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:48.994060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:42059)
0928 06:33:48.994177 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:48.994221 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 06:33:49.117197 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.116975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:33377)
0928 06:33:49.117110 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:49.117152 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260928 06:33:49.913762 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.913487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:34187)
0928 06:33:49.913668 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:49.913709 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":119,"threads_started":1}
I20260928 06:33:49.994985 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:49.994768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:56157)
0928 06:33:49.994904 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:49.994938 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260928 06:33:50.117974 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.117744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:58493)
0928 06:33:50.117886 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.117928 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260928 06:33:50.793504 7475 raft_consensus.cc:482] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:50.793685 7475 raft_consensus.cc:504] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db2c874a4df04beab836e8db1a1997c0" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 37215 } } peers { permanent_uuid: "39125b0dd79345ed9279d17c9f558ac8" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39625 } } peers { permanent_uuid: "f2878ceab4bb467cafddb7223a4e709c" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 37613 } }
I20260928 06:33:50.794034 7475 leader_election.cc:288] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215), f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613)
W20260928 06:33:50.794409 7052 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db2c874a4df04beab836e8db1a1997c0 (127.24.11.131:37215): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
W20260928 06:33:50.794461 7053 leader_election.cc:341] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f2878ceab4bb467cafddb7223a4e709c (127.24.11.130:37613): Not found: Tablet not found: 80176ac3b1204fafa116ec7f9085710b
I20260928 06:33:50.794488 7052 leader_election.cc:302] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39125b0dd79345ed9279d17c9f558ac8; no voters: db2c874a4df04beab836e8db1a1997c0, f2878ceab4bb467cafddb7223a4e709c
I20260928 06:33:50.794548 7475 raft_consensus.cc:2726] T 80176ac3b1204fafa116ec7f9085710b P 39125b0dd79345ed9279d17c9f558ac8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:33:50.914427 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.914195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:50277)
0928 06:33:50.914333 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.914371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260928 06:33:50.995728 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:50.995500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:49373)
0928 06:33:50.995641 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:50.995683 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260928 06:33:51.118664 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.118427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:38877)
0928 06:33:51.118581 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.118620 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260928 06:33:51.915187 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.914961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:55737)
0928 06:33:51.915099 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.915134 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 06:33:51.996481 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:51.996256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:39443)
0928 06:33:51.996389 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:51.996435 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260928 06:33:52.119417 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:52.119166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:43251)
0928 06:33:52.119310 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:52.119369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260928 06:33:52.915871 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:52.915656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:56541)
0928 06:33:52.915781 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:52.915814 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 06:33:52.997267 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:52.997032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:37961)
0928 06:33:52.997175 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:52.997222 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 06:33:53.120229 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:53.119958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:35767)
0928 06:33:53.120130 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:53.120180 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260928 06:33:53.916643 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:53.916436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.129:58405)
0928 06:33:53.916561 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:53.916600 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260928 06:33:53.998091 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:53.997805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.130:51477)
0928 06:33:53.998009 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:53.998051 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260928 06:33:54.117734 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7005
I20260928 06:33:54.121006 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:54.120808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:42751 (local address 127.24.11.131:59423)
0928 06:33:54.120933 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:54.120967 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:42751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260928 06:33:54.122618 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7169
I20260928 06:33:54.127000 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7302
2026-09-28T06:33:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18541 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:33:54 dist-test-slave-38pg krb5kdc[7490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:33:54 dist-test-slave-38pg krb5kdc[7490](info): set up 2 sockets
Sep 28 06:33:54 dist-test-slave-38pg krb5kdc[7490](info): commencing operation
krb5kdc: starting...
W20260928 06:33:55.162750 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:33:55.280153 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:55.361236 7503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.361498 7503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.361541 7503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.362741 7503 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:33:55.362810 7503 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:33:55.362831 7503 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:33:55.362850 7503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:33:55.362880 7503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.362905 7503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.362936 7503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:33:55.362964 7503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:33:55.364445 7503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:55.364727 7503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.364951 7503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.369867 7509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.369867 7511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.369871 7508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.370115 7503 server_base.cc:1034] running on GCE node
I20260928 06:33:55.370342 7503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.370584 7503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.371727 7503 hybrid_clock.cc:648] HybridClock initialized: now 1790577235371716 us; error 27 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.374337 7503 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:33:55.374814 7503 webserver.cc:460] Webserver started at http://127.24.11.190:34387/ using document root <none> and password file <none>
I20260928 06:33:55.375049 7503 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.375118 7503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.375269 7503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:55.376338 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "307dba354c364493be9c978b003b7c90"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "ad879a04d6d94e5087fd735f224e12fc"
server_key_iv: "f4187483d3b8b5323fa482397d2aeee7"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.376710 7503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "307dba354c364493be9c978b003b7c90"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "ad879a04d6d94e5087fd735f224e12fc"
server_key_iv: "f4187483d3b8b5323fa482397d2aeee7"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.378129 7503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:33:55.378944 7518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.379158 7503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:33:55.379247 7503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "307dba354c364493be9c978b003b7c90"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "ad879a04d6d94e5087fd735f224e12fc"
server_key_iv: "f4187483d3b8b5323fa482397d2aeee7"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.379320 7503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.389978 7503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.391727 7503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.391903 7503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.395735 7503 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:36965
I20260928 06:33:55.395772 7570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:36965 every 8 connection(s)
I20260928 06:33:55.396149 7503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:33:55.397011 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:55.399298 7571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap starting.
I20260928 06:33:55.399828 7571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:55.400076 7571 log.cc:824] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:55.400710 7571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: No bootstrap required, opened a new log
I20260928 06:33:55.402321 7571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:33:55.402457 7571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:55.402480 7571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307dba354c364493be9c978b003b7c90, State: Initialized, Role: FOLLOWER
I20260928 06:33:55.402621 7571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:33:55.402704 7571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:33:55.402731 7571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:33:55.402760 7571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:55.403239 7571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:33:55.403321 7571 leader_election.cc:302] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307dba354c364493be9c978b003b7c90; no voters:
I20260928 06:33:55.403450 7571 leader_election.cc:288] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:33:55.403522 7574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:55.403700 7574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 LEADER]: Becoming Leader. State: Replica: 307dba354c364493be9c978b003b7c90, State: Running, Role: LEADER
I20260928 06:33:55.403712 7571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:33:55.403800 7574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:33:55.404402 7575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:33:55.404425 7576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307dba354c364493be9c978b003b7c90. Latest consensus state: current_term: 1 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:33:55.404511 7575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:55.404528 7576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:33:55.404745 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7503
I20260928 06:33:55.404853 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:33:55.405084 24622 external_mini_cluster.cc:1428] Setting key 87adb02efcf3647aadd75975086438d6
W20260928 06:33:55.405889 7589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:33:55.405944 7589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:33:55.405995 7583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:33:55.406160 7583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:33:55.410166 7583 catalog_manager.cc:1347] Generated new cluster ID: 7991a185025543188ea00d5165a3432f
I20260928 06:33:55.410229 7583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:55.415989 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.405821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.24.11.190:36965)
0928 06:33:55.406074 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.406084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.406115 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:33:55.406230 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.406236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.406289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.406425 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.407154 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.408183 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.408955 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.409387 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.409501 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.412360 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.412364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.412364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.412376 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.414344 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.414764 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.414767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.414767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.414869 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.415079 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.415079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.415080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.415370 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.415510 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.415634 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.415761 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260928 06:33:55.431717 7583 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:33:55.432466 7583 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:33:55.441107 7583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Generated new TSK 0
I20260928 06:33:55.441299 7583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:33:55.468686 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:55.550592 7597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.550817 7597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.550856 7597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.552002 7597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.552067 7597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.552112 7597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:33:55.553623 7597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:55.553872 7597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.554105 7597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.559175 7605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.559190 7603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.559190 7597 server_base.cc:1034] running on GCE node
W20260928 06:33:55.559162 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.559504 7597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.559770 7597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.560915 7597 hybrid_clock.cc:648] HybridClock initialized: now 1790577235560908 us; error 50 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.563441 7597 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:33:55.563815 7597 webserver.cc:460] Webserver started at http://127.24.11.129:33127/ using document root <none> and password file <none>
I20260928 06:33:55.564010 7597 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.564049 7597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.564136 7597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:55.565112 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "0492e01a443074ef31e216ca009e4253"
server_key_iv: "94af8bdf543822ccd47d4b1e79455535"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.565408 7597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "0492e01a443074ef31e216ca009e4253"
server_key_iv: "94af8bdf543822ccd47d4b1e79455535"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.566847 7597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:55.567565 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.567752 7597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:33:55.567842 7597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "0492e01a443074ef31e216ca009e4253"
server_key_iv: "94af8bdf543822ccd47d4b1e79455535"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.567917 7597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.591892 7597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.594009 7597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.594162 7597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.594410 7597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:55.594730 7597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:55.594765 7597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.594787 7597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:55.594832 7597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.600559 7597 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:45907
I20260928 06:33:55.600653 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:45907 every 8 connection(s)
I20260928 06:33:55.600966 7597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:33:55.603823 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7597
I20260928 06:33:55.603924 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:33:55.604117 24622 external_mini_cluster.cc:1428] Setting key 2eb8ca306e1a5ec51bc83ce02ab46879
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:55.611277 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.601799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33953 (local address 127.24.11.190:36965)
0928 06:33:55.601863 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.601866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.602062 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:33:55.602601 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.602602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.602619 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.602667 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.603656 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.604393 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.605611 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.605832 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.605925 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.608128 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.608129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.608130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.608139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.609879 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.610292 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.610293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.610293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.610361 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.610602 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.610603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.610603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.610749 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.610827 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.611076 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.611170 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:33:55.611306 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.601775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:33953)
0928 06:33:55.601940 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.601976 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.602492 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.602763 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.602767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.603338 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.603535 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.603540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.604505 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.604509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.605482 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.605485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.605937 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.605938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.606073 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.606631 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.606640 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.608030 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:55.609982 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.609988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.609990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.610210 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.610438 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.610438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.610439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.610536 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.610819 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:55.610825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:55.610942 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.611026 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.611148 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260928 06:33:55.611860 7726 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:33:55.611965 7726 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:55.612166 7726 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:33:55.612718 7535 ts_manager.cc:194] Registered new tserver with Master: 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129:45907)
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
I20260928 06:33:55.613381 7535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:33953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:33:55.630580 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:55.711455 7731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.711707 7731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.711750 7731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.712889 7731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.712954 7731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.713001 7731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:33:55.714468 7731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:55.714766 7731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.714993 7731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.720062 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.720141 7731 server_base.cc:1034] running on GCE node
W20260928 06:33:55.720077 7739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.720078 7736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.720485 7731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.720726 7731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.721882 7731 hybrid_clock.cc:648] HybridClock initialized: now 1790577235721863 us; error 36 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.724387 7731 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:33:55.724884 7731 webserver.cc:460] Webserver started at http://127.24.11.130:41209/ using document root <none> and password file <none>
I20260928 06:33:55.725103 7731 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.725169 7731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.725319 7731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:55.726428 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "663f7e36c6574609bc20595718abb43f"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "5eb78c8e2c3c15a3f47d8d6aa3a52c73"
server_key_iv: "c6e6f44fb848e75e450e8833003982fa"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.726797 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "663f7e36c6574609bc20595718abb43f"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "5eb78c8e2c3c15a3f47d8d6aa3a52c73"
server_key_iv: "c6e6f44fb848e75e450e8833003982fa"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.728124 7731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:33:55.728820 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.728977 7731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:33:55.729084 7731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "663f7e36c6574609bc20595718abb43f"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "5eb78c8e2c3c15a3f47d8d6aa3a52c73"
server_key_iv: "c6e6f44fb848e75e450e8833003982fa"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.729158 7731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.739886 7731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.742434 7731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.742604 7731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.742807 7731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:55.743134 7731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:55.743168 7731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.743212 7731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:55.743238 7731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.749069 7731 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:44387
I20260928 06:33:55.749151 7859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:44387 every 8 connection(s)
I20260928 06:33:55.749477 7731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:33:55.755188 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7731
I20260928 06:33:55.755312 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:33:55.755559 24622 external_mini_cluster.cc:1428] Setting key 749da6a406163f89de57a740898f0659
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:55.760310 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.750286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:43591 (local address 127.24.11.190:36965)
0928 06:33:55.750337 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.750339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.750532 (+ 193us) server_negotiation.cc:407] Connection header received
0928 06:33:55.751088 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.751090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.751103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.751161 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.752113 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.752640 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.753722 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.753940 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.754026 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.756707 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.756708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.756708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.756716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.758776 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.759307 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.759308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.759309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.759373 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.759641 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.759642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.759643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.759765 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.759840 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.760118 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.760207 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:33:55.760512 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.750245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:43591)
0928 06:33:55.750405 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.750446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.750979 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.751235 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.751239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.751815 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.752013 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.752018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.752748 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.752749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.753637 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.753640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.754037 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.754038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.754165 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.754727 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.754736 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.756547 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:55.758883 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.758891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.758893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.759207 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.759441 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.759441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.759442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.759559 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.759830 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:55.759836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:55.759984 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.760097 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.760264 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 06:33:55.760941 7860 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:33:55.761029 7860 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:55.761169 7860 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:33:55.761510 7535 ts_manager.cc:194] Registered new tserver with Master: 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:33:55.762029 7535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:43591
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:33:55.792243 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:33:55.878443 7865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:33:55.878641 7865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:33:55.878665 7865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:33:55.879788 7865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:33:55.879822 7865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:33:55.879848 7865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:33:55.881222 7865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:33:55.881407 7865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:33:55.881661 7865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:33:55.886670 7873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.886705 7870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:33:55.886719 7871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:33:55.886936 7865 server_base.cc:1034] running on GCE node
I20260928 06:33:55.887115 7865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:33:55.887329 7865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:33:55.888484 7865 hybrid_clock.cc:648] HybridClock initialized: now 1790577235888461 us; error 35 us; skew 500 ppm
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:33:55.891104 7865 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:33:55.891484 7865 webserver.cc:460] Webserver started at http://127.24.11.131:36781/ using document root <none> and password file <none>
I20260928 06:33:55.891678 7865 fs_manager.cc:359] Metadata directory not provided
I20260928 06:33:55.891719 7865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:33:55.891806 7865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:33:55.892866 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "9010f7b134564a7b839849bd6243eb38"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "f75b4cac2cbbee2e15687b278fc14d15"
server_key_iv: "d8e6b14d12448c9b3d3ff81a6c4bae3e"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.893189 7865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "9010f7b134564a7b839849bd6243eb38"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "f75b4cac2cbbee2e15687b278fc14d15"
server_key_iv: "d8e6b14d12448c9b3d3ff81a6c4bae3e"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.894577 7865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:33:55.895321 7880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.895644 7865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:33:55.895736 7865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "9010f7b134564a7b839849bd6243eb38"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "f75b4cac2cbbee2e15687b278fc14d15"
server_key_iv: "d8e6b14d12448c9b3d3ff81a6c4bae3e"
server_key_version: "encryptionkey@0"
I20260928 06:33:55.895794 7865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:33:55.916966 7865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:33:55.919061 7865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:33:55.919178 7865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:33:55.919384 7865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:33:55.919715 7865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:33:55.919751 7865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.919799 7865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:33:55.919829 7865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:33:55.926203 7865 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:41661
I20260928 06:33:55.926254 7993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:41661 every 8 connection(s)
I20260928 06:33:55.926590 7865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:33:55.927485 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 7865
I20260928 06:33:55.927577 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:33:55.927802 24622 external_mini_cluster.cc:1428] Setting key dd7166860691c4043f42510da5eb673f
Sep 28 06:33:55 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577235, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:33:55.936662 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.927352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40301 (local address 127.24.11.190:36965)
0928 06:33:55.927414 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.927416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.927682 (+ 266us) server_negotiation.cc:407] Connection header received
0928 06:33:55.928443 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.928444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.928457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.928522 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.929525 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.930103 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.931094 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.931286 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.931345 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.933743 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.933745 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.933745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.933751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.935214 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.935624 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.935625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.935625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.935682 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.936023 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.936024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.936025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.936160 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.936238 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.936528 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.936597 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:33:55.936756 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.927362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:40301)
0928 06:33:55.927538 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.927600 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.928312 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.928594 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.928598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.929246 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:33:55.929444 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.929449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.930191 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.930192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.931016 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.931018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.931363 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.931364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.931476 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.932066 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:33:55.932074 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:33:55.933620 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:33:55.935315 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.935322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.935323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.935545 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.935807 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:33:55.935808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:33:55.935808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:33:55.935922 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:33:55.936239 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:33:55.936244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:33:55.936365 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.936461 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.936589 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260928 06:33:55.937256 7994 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:33:55.937342 7994 heartbeater.cc:464] Registering TS with master...
I20260928 06:33:55.937588 7994 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:33:55.938026 7535 ts_manager.cc:194] Registered new tserver with Master: 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661)
I20260928 06:33:55.938441 7535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:40301
I20260928 06:33:55.939730 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:33:55.947340 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.941173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.24.11.190:36965)
0928 06:33:55.941226 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.941228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.941310 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:33:55.941363 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.941363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.941372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.941421 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:33:55.942008 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.942505 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.943294 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.943596 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.943656 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.944735 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:33:55.944736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:33:55.944736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:33:55.944742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:33:55.946306 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.946607 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.946608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.946608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.946660 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:33:55.946868 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:33:55.946869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:33:55.946869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:33:55.947013 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:33:55.947096 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.947175 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.947234 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:33:55.949182 7535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:33:55.950938 7535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:33:55.957557 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51178 (local address 127.24.11.130:44387)
0928 06:33:55.953835 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.953839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.954155 (+ 316us) server_negotiation.cc:407] Connection header received
0928 06:33:55.954202 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.954204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.954220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.954298 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:55.954452 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.955479 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.956600 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957100 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957198 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957261 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.957417 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.957458 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260928 06:33:55.957523 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.0.0.1:51178)
0928 06:33:55.954035 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.954045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.954093 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.954317 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.954318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.954320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:55.954368 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.954369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.955578 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.955579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.956513 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.956515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957202 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.957279 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957292 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.957393 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.957447 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":100,"threads_started":1}
I20260928 06:33:55.958091 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.24.11.131:41661)
0928 06:33:55.953520 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.953547 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.953560 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:33:55.953604 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.953607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.953625 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.953722 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:55.953982 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.954956 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.956594 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957414 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957581 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.957936 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.957994 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
I20260928 06:33:55.958082 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41661 (local address 127.0.0.1:43206)
0928 06:33:55.953246 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.953286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.953344 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.953807 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.953810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.953812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:55.953886 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.953889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.955063 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.955064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.956486 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.956489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957620 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.957693 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957829 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.957888 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.957990 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":45,"thread_start_us":56,"threads_started":1}
I20260928 06:33:55.958451 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.24.11.129:45907)
0928 06:33:55.953749 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.953755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.953766 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:55.953811 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.953813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.953829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.953923 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:55.954083 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.955256 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.956653 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957441 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957504 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957594 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.958128 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.958186 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":117,"threads_started":1}
I20260928 06:33:55.958407 7794 tablet_service.cc:1467] Processing CreateTablet for tablet ff8416fea2b34c42b14b34a9456eb996 (DEFAULT_TABLE table=test-table [id=95661e5d32cd43d496579b755967ba36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:55.958631 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.953341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45907 (local address 127.0.0.1:55068)
0928 06:33:55.953682 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.953694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.953741 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.953938 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.953939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.953942 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:55.953987 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.953989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.955375 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.955376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.956560 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.956562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.957818 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.957819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.957925 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.957930 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.957981 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.958024 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":3,"thread_start_us":85,"threads_started":1}
I20260928 06:33:55.958791 7794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8416fea2b34c42b14b34a9456eb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:55.959115 7928 tablet_service.cc:1467] Processing CreateTablet for tablet ff8416fea2b34c42b14b34a9456eb996 (DEFAULT_TABLE table=test-table [id=95661e5d32cd43d496579b755967ba36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:55.959378 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8416fea2b34c42b14b34a9456eb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:55.959307 7660 tablet_service.cc:1467] Processing CreateTablet for tablet ff8416fea2b34c42b14b34a9456eb996 (DEFAULT_TABLE table=test-table [id=95661e5d32cd43d496579b755967ba36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:33:55.959628 7660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8416fea2b34c42b14b34a9456eb996. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:33:55.962209 8011 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Bootstrap starting.
I20260928 06:33:55.962441 8012 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Bootstrap starting.
I20260928 06:33:55.962917 8013 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap starting.
I20260928 06:33:55.963109 8012 tablet_bootstrap.cc:654] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:55.963124 8011 tablet_bootstrap.cc:654] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:55.963532 8012 log.cc:824] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:55.963812 8011 log.cc:824] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:55.963879 8013 tablet_bootstrap.cc:654] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Neither blocks nor log segments found. Creating new log.
I20260928 06:33:55.964319 8012 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: No bootstrap required, opened a new log
I20260928 06:33:55.964416 8012 ts_tablet_manager.cc:1397] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:33:55.964461 8013 log.cc:824] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:33:55.964941 8011 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: No bootstrap required, opened a new log
I20260928 06:33:55.965015 8011 ts_tablet_manager.cc:1397] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:33:55.965262 8013 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: No bootstrap required, opened a new log
I20260928 06:33:55.965335 8013 ts_tablet_manager.cc:1397] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:55.966539 8012 raft_consensus.cc:348] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.966744 8012 raft_consensus.cc:374] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:55.966782 8012 raft_consensus.cc:729] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f7e36c6574609bc20595718abb43f, State: Initialized, Role: FOLLOWER
I20260928 06:33:55.966724 8011 raft_consensus.cc:348] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.966908 8011 raft_consensus.cc:374] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:55.966953 8011 raft_consensus.cc:729] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9010f7b134564a7b839849bd6243eb38, State: Initialized, Role: FOLLOWER
I20260928 06:33:55.966936 8012 consensus_queue.cc:260] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.967105 8011 consensus_queue.cc:260] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.967219 8012 ts_tablet_manager.cc:1428] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:55.967348 7860 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
I20260928 06:33:55.967425 8011 ts_tablet_manager.cc:1428] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:33:55.967351 8013 raft_consensus.cc:348] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.967486 7994 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
I20260928 06:33:55.967507 8013 raft_consensus.cc:374] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:33:55.967541 8013 raft_consensus.cc:729] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Initialized, Role: FOLLOWER
I20260928 06:33:55.967645 8013 consensus_queue.cc:260] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.967904 8013 ts_tablet_manager.cc:1428] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:33:55.968029 7726 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
I20260928 06:33:55.982162 8019 raft_consensus.cc:482] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:33:55.982283 8019 raft_consensus.cc:504] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.982630 8019 leader_election.cc:288] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:33:55.985478 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.982907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:39377 (local address 127.24.11.131:41661)
0928 06:33:55.982977 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.982980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.982989 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:33:55.983022 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.983022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.983033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.983098 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:55.983267 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.983771 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.984735 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.985113 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.985197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.985206 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.985365 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.985413 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:33:55.985478 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.982805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41661 (local address 127.24.11.129:39377)
0928 06:33:55.982844 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.982857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.982901 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.983106 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.983107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.983109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:55.983191 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.983193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.983865 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.983866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.984628 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.984630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.985194 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.985196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.985276 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.985280 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.985356 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.985399 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 06:33:55.985845 7948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8416fea2b34c42b14b34a9456eb996" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9010f7b134564a7b839849bd6243eb38" is_pre_election: true
I20260928 06:33:55.985888 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.983031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:53015 (local address 127.24.11.130:44387)
0928 06:33:55.983115 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:55.983117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:55.983245 (+ 128us) server_negotiation.cc:407] Connection header received
0928 06:33:55.983336 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:55.983337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:55.983348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:55.983409 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:33:55.983580 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.984049 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.984983 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.985369 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.985449 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.985457 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:55.985782 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 06:33:55.985821 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:33:55.985901 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:55.982955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.24.11.129:53015)
0928 06:33:55.983168 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:55.983177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:33:55.983255 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:33:55.983416 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:33:55.983417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:33:55.983419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:33:55.983501 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.983503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.984133 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.984134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.984889 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:33:55.984891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:33:55.985461 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:33:55.985462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:33:55.985689 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:55.985695 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:33:55.985769 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:33:55.985812 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
I20260928 06:33:55.986034 7948 raft_consensus.cc:2454] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 0.
I20260928 06:33:55.986218 7814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8416fea2b34c42b14b34a9456eb996" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f7e36c6574609bc20595718abb43f" is_pre_election: true
I20260928 06:33:55.986292 7615 leader_election.cc:302] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, 9010f7b134564a7b839849bd6243eb38; no voters:
I20260928 06:33:55.986393 7814 raft_consensus.cc:2454] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 0.
I20260928 06:33:55.986416 8019 raft_consensus.cc:2781] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:33:55.986480 8019 raft_consensus.cc:482] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:33:55.986518 8019 raft_consensus.cc:3037] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:55.987249 8019 raft_consensus.cc:504] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.987391 8019 leader_election.cc:288] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 election: Requested vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:33:55.987574 7948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8416fea2b34c42b14b34a9456eb996" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9010f7b134564a7b839849bd6243eb38"
I20260928 06:33:55.987584 7814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8416fea2b34c42b14b34a9456eb996" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f7e36c6574609bc20595718abb43f"
I20260928 06:33:55.987650 7948 raft_consensus.cc:3037] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:55.987672 7814 raft_consensus.cc:3037] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:33:55.988337 7948 raft_consensus.cc:2454] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 1.
I20260928 06:33:55.988525 7814 raft_consensus.cc:2454] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 1.
I20260928 06:33:55.988570 7615 leader_election.cc:302] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, 9010f7b134564a7b839849bd6243eb38; no voters:
I20260928 06:33:55.988688 8019 raft_consensus.cc:2781] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:33:55.988891 8019 raft_consensus.cc:686] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 LEADER]: Becoming Leader. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Running, Role: LEADER
I20260928 06:33:55.989004 8019 consensus_queue.cc:237] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:33:55.989910 7534 catalog_manager.cc:5495] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129). New cstate: current_term: 1 leader_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } health_report { overall_health: UNKNOWN } } }
W20260928 06:33:56.000062 7861 tablet.cc:2369] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:33:56.003064 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.000879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.24.11.129:45907)
0928 06:33:56.000919 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:33:56.000921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:33:56.000932 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:33:56.000999 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:33:56.001000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:33:56.001016 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:33:56.001080 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:33:56.001176 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.001712 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.002424 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:33:56.002610 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:33:56.002663 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:33:56.002785 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:33:56.002882 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:33:56.002941 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:33:56.002976 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:33:56.002992 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:33:56.005100 7814 raft_consensus.cc:1261] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Refusing update from remote peer 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:56.005101 7948 raft_consensus.cc:1261] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Refusing update from remote peer 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:33:56.005558 8021 consensus_queue.cc:1035] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:56.005710 8019 consensus_queue.cc:1035] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:33:56.008111 8024 mvcc.cc:205] Tried to move back new op lower bound from 7334204358673551360 to 7334204358611685376. Current Snapshot: MvccSnapshot[applied={T|T < 7334204358673551360}]
I20260928 06:33:56.008176 8025 mvcc.cc:205] Tried to move back new op lower bound from 7334204358673551360 to 7334204358611685376. Current Snapshot: MvccSnapshot[applied={T|T < 7334204358673551360}]
I20260928 06:33:56.008669 8026 mvcc.cc:205] Tried to move back new op lower bound from 7334204358673551360 to 7334204358611685376. Current Snapshot: MvccSnapshot[applied={T|T < 7334204358673551360}]
I20260928 06:33:56.010679 7534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:56.010764 7534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:33:56.011561 7534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Sending DeleteTablet for 3 replicas of tablet ff8416fea2b34c42b14b34a9456eb996
I20260928 06:33:56.011910 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7597
I20260928 06:33:56.011989 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8416fea2b34c42b14b34a9456eb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:56 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43206
I20260928 06:33:56.011991 7794 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8416fea2b34c42b14b34a9456eb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:56 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:51178
I20260928 06:33:56.017326 8034 tablet_replica.cc:330] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: stopping tablet replica
W20260928 06:33:56.017356 7521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:45907 (error 108)
I20260928 06:33:56.017452 8034 raft_consensus.cc:2229] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:56.017597 8034 raft_consensus.cc:2258] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:56.017670 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7731
I20260928 06:33:56.017719 8035 tablet_replica.cc:330] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: stopping tablet replica
I20260928 06:33:56.017828 8035 raft_consensus.cc:2229] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:33:56.017917 8035 raft_consensus.cc:2258] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:33:56.018469 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.018333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45907 (local address 127.0.0.1:55084)
0928 06:33:56.018392 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:56.018434 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:33:56.018558 8035 ts_tablet_manager.cc:1905] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:33:56.018582 7521 catalog_manager.cc:4584] TS 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129:45907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff8416fea2b34c42b14b34a9456eb996: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45907: connect: Connection refused (error 111)
I20260928 06:33:56.020572 8035 ts_tablet_manager.cc:1918] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:33:56.020653 8035 log.cc:1196] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/ff8416fea2b34c42b14b34a9456eb996
I20260928 06:33:56.020937 8035 ts_tablet_manager.cc:1939] T ff8416fea2b34c42b14b34a9456eb996 P 9010f7b134564a7b839849bd6243eb38: Deleting consensus metadata
I20260928 06:33:56.021376 7521 catalog_manager.cc:4841] TS 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): tablet ff8416fea2b34c42b14b34a9456eb996 (table test-table [id=95661e5d32cd43d496579b755967ba36]) successfully deleted
I20260928 06:33:56.023358 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7865
I20260928 06:33:56.023409 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:33:56.023291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.0.0.1:51184)
0928 06:33:56.023345 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:33:56.023371 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:44387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:33:56.027891 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 7503
I20260928 06:34:16.033282 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--webserver_interface=127.24.11.190
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:16.114979 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:16.115229 8037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:16.115270 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:16.116428 8037 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:34:16.116492 8037 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:34:16.116513 8037 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:34:16.116530 8037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:16.116559 8037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:16.116588 8037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:16.116616 8037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:16.116645 8037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:16.118156 8037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:16.118443 8037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:16.118670 8037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:16.128012 8044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.128062 8037 server_base.cc:1034] running on GCE node
W20260928 06:34:16.128036 8043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:16.128022 8046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.128645 8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:16.128955 8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:16.130092 8037 hybrid_clock.cc:648] HybridClock initialized: now 1790577256130080 us; error 30 us; skew 500 ppm
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:16.139591 8037 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:34:16.140146 8037 webserver.cc:460] Webserver started at http://127.24.11.190:34387/ using document root <none> and password file <none>
I20260928 06:34:16.140394 8037 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:16.140482 8037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:16.141978 8037 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:34:16.142683 8053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.142968 8037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:16.143069 8037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "307dba354c364493be9c978b003b7c90"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "ad879a04d6d94e5087fd735f224e12fc"
server_key_iv: "f4187483d3b8b5323fa482397d2aeee7"
server_key_version: "encryptionkey@0"
I20260928 06:34:16.143488 8037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:16.159739 8037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:16.162145 8037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:16.162322 8037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:16.166628 8037 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:36965
I20260928 06:34:16.166699 8105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:36965 every 8 connection(s)
I20260928 06:34:16.167057 8037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:16.168565 8106 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:34:16.168619 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8037
I20260928 06:34:16.168895 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:45907
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=33127
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:34:16.169339 8106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap starting.
I20260928 06:34:16.178427 8106 log.cc:824] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:16.180369 8106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:34:16.180675 8106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap complete.
I20260928 06:34:16.183044 8106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:16.183290 8106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307dba354c364493be9c978b003b7c90, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.183465 8106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:16.183544 8106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:16.183601 8106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:16.183661 8106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:34:16.184680 8106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:16.184814 8106 leader_election.cc:302] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307dba354c364493be9c978b003b7c90; no voters:
I20260928 06:34:16.184998 8106 leader_election.cc:288] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:34:16.185034 8110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:34:16.185153 8110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 LEADER]: Becoming Leader. State: Replica: 307dba354c364493be9c978b003b7c90, State: Running, Role: LEADER
I20260928 06:34:16.185268 8106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:16.185248 8110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:16.185544 8110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307dba354c364493be9c978b003b7c90. Latest consensus state: current_term: 2 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:34:16.185611 8110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:16.185861 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:34:16.185942 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:16.186465 8114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:16.187269 8114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:16.187379 8114 catalog_manager.cc:1259] Loaded cluster ID: 7991a185025543188ea00d5165a3432f
I20260928 06:34:16.187410 8114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:16.188608 8114 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:16.188784 8114 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Loaded TSK: 0
I20260928 06:34:16.215396 8114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Generated new TSK 1
I20260928 06:34:16.215519 8114 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:34:16.259748 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:16.259999 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:16.260025 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:16.261216 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:16.261262 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:16.261289 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:16.262844 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:45907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=33127
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:16.263092 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:16.263334 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:16.268383 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:16.268383 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:16.268585 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.268596 8108 server_base.cc:1034] running on GCE node
I20260928 06:34:16.268880 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:16.269150 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:16.270318 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790577256270295 us; error 38 us; skew 500 ppm
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:16.272796 8108 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:34:16.273288 8108 webserver.cc:460] Webserver started at http://127.24.11.129:33127/ using document root <none> and password file <none>
I20260928 06:34:16.273576 8108 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:16.273643 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:16.275058 8108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:16.275751 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.275902 8108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:16.276000 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "0492e01a443074ef31e216ca009e4253"
server_key_iv: "94af8bdf543822ccd47d4b1e79455535"
server_key_version: "encryptionkey@0"
I20260928 06:34:16.276367 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:16.287531 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:16.289866 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:16.290014 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:16.290267 8108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:16.290728 8150 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:34:16.292143 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:34:16.292194 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:34:16.292220 8108 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:34:16.292873 8108 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:34:16.292917 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:16.292965 8150 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap starting.
I20260928 06:34:16.300876 8108 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:45907
I20260928 06:34:16.301025 8257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:45907 every 8 connection(s)
I20260928 06:34:16.301404 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:16.303936 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8108
I20260928 06:34:16.304221 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:44387
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=41209
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:34:16.304778 8150 log.cc:824] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:16.307389 8150 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:34:16.307693 8150 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap complete.
I20260928 06:34:16.308956 8150 ts_tablet_manager.cc:1397] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260928 06:34:16.310417 8150 raft_consensus.cc:348] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.310729 8150 raft_consensus.cc:729] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.310887 8150 consensus_queue.cc:260] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.311131 8150 ts_tablet_manager.cc:1428] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:16.318272 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.302128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:40315)
0928 06:34:16.305064 (+ 2936us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.305111 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.308149 (+ 3038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.308766 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.308770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.309218 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:16.309494 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.309501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.311350 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.311351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.312131 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.312133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.312727 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.312728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.312854 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.313402 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:16.313410 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:16.314792 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:16.316633 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.316640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.316642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.316870 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.317126 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.317127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.317128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.317231 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.317650 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:16.317656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:16.317794 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.317897 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.318072 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"thread_start_us":94,"threads_started":1}
I20260928 06:34:16.318272 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.302264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:40315 (local address 127.24.11.190:36965)
0928 06:34:16.302454 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.302464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.305639 (+ 3175us) server_negotiation.cc:407] Connection header received
0928 06:34:16.308444 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.308451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.308514 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.308632 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.309677 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.311260 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.312237 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.312622 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.312750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.314910 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.314918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.314919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.314930 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.316530 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.316958 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.316961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.316962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.317053 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.317300 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.317301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.317301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.317520 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.317670 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.317963 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.318090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260928 06:34:16.319289 8258 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:34:16.319387 8258 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:16.319559 8258 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:34:16.320220 8070 ts_manager.cc:194] Registered new tserver with Master: 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129:45907)
I20260928 06:34:16.321278 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:40315
I20260928 06:34:16.322342 8258 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
I20260928 06:34:16.325292 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.320805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45907 (local address 127.0.0.1:36132)
0928 06:34:16.321767 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.321804 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.321868 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.322325 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.322328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.322330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:16.322392 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.322394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.323463 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.323464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.324334 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.324337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.324978 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.324979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.325058 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.325070 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.325165 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.325222 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":101,"threads_started":1}
I20260928 06:34:16.325292 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.321595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.24.11.129:45907)
0928 06:34:16.321988 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.321993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.322001 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:16.322030 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.322032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.322178 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.322255 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:16.322475 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.323381 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.324423 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.324883 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.324974 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.325030 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.325183 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.325222 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":86,"threads_started":1}
I20260928 06:34:16.325685 8192 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8416fea2b34c42b14b34a9456eb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:56 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:36132
I20260928 06:34:16.326174 8267 tablet_replica.cc:330] stopping tablet replica
I20260928 06:34:16.326277 8267 raft_consensus.cc:2229] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:16.326354 8267 raft_consensus.cc:2258] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:16.326650 8267 ts_tablet_manager.cc:1905] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:16.328830 8267 ts_tablet_manager.cc:1918] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:16.328899 8267 log.cc:1196] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal/wals/ff8416fea2b34c42b14b34a9456eb996
I20260928 06:34:16.329136 8267 ts_tablet_manager.cc:1939] T ff8416fea2b34c42b14b34a9456eb996 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Deleting consensus metadata
I20260928 06:34:16.329728 8056 catalog_manager.cc:4841] TS 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129:45907): tablet ff8416fea2b34c42b14b34a9456eb996 (table test-table [id=95661e5d32cd43d496579b755967ba36]) successfully deleted
W20260928 06:34:16.394948 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:16.395167 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:16.395192 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:16.396448 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:16.396497 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:16.396526 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:16.398279 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:44387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=41209
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:16.398489 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:16.398684 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:16.403961 8272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.404059 8263 server_base.cc:1034] running on GCE node
W20260928 06:34:16.403977 8273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:16.403977 8275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.404415 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:16.404675 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:16.405874 8263 hybrid_clock.cc:648] HybridClock initialized: now 1790577256405855 us; error 43 us; skew 500 ppm
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:16.408855 8263 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:34:16.409286 8263 webserver.cc:460] Webserver started at http://127.24.11.130:41209/ using document root <none> and password file <none>
I20260928 06:34:16.409487 8263 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:16.409605 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:16.410987 8263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:16.411628 8282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.411829 8263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:16.411911 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "663f7e36c6574609bc20595718abb43f"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "5eb78c8e2c3c15a3f47d8d6aa3a52c73"
server_key_iv: "c6e6f44fb848e75e450e8833003982fa"
server_key_version: "encryptionkey@0"
I20260928 06:34:16.412240 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:16.425508 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:16.427711 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:16.427865 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:16.428061 8263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:16.428521 8289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:34:16.430071 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:34:16.430122 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:34:16.430153 8263 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:34:16.430862 8263 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:34:16.430897 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:16.431008 8289 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Bootstrap starting.
I20260928 06:34:16.437757 8263 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:44387
I20260928 06:34:16.437839 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:44387 every 8 connection(s)
I20260928 06:34:16.438169 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:16.440817 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8263
I20260928 06:34:16.441093 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:41661
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=36781
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:34:16.441515 8289 log.cc:824] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:16.445060 8289 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:16.445408 8289 tablet_bootstrap.cc:492] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Bootstrap complete.
I20260928 06:34:16.446630 8289 ts_tablet_manager.cc:1397] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260928 06:34:16.448035 8289 raft_consensus.cc:348] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.448426 8289 raft_consensus.cc:729] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f7e36c6574609bc20595718abb43f, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.448565 8289 consensus_queue.cc:260] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.448906 8289 ts_tablet_manager.cc:1428] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:34:16.449085 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.438928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:54375 (local address 127.24.11.190:36965)
0928 06:34:16.438998 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.439001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.439162 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:34:16.439685 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.439686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.439702 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.439763 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.440382 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.441068 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.442999 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.443328 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.443408 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.445951 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.445953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.445953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.445961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.447604 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.448038 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.448039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.448040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.448104 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.448351 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.448351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.448352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.448502 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.448577 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.448893 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.448994 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:34:16.449306 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.438848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:54375)
0928 06:34:16.439030 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.439073 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.439579 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.439820 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.439825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.440078 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:16.440284 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.440289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.441803 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.441804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.442884 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.442887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.443450 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.443451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.443604 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.444175 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:16.444184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:16.445849 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:16.447713 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.447720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.447722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.447960 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.448177 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.448178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.448178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.448286 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.448576 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:16.448583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:16.448746 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.448870 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.449036 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260928 06:34:16.449725 8397 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:34:16.449817 8397 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:16.449954 8397 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:34:16.450452 8070 ts_manager.cc:194] Registered new tserver with Master: 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:34:16.451045 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:54375
I20260928 06:34:16.451712 8397 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.130@KRBTEST.COM
I20260928 06:34:16.459898 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.450786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.0.0.1:57978)
0928 06:34:16.450842 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.450853 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.450912 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.451217 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.451218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.451662 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:16.451717 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.451719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.453150 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.453151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.453744 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.453747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.454437 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.454438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.454484 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.455036 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:16.455042 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:16.456606 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:16.458739 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.458747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.458749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.458962 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.459197 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.459197 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.459198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.459267 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.459573 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:16.459579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:16.459627 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.459689 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.459828 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:34:16.459925 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.450858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.24.11.130:44387)
0928 06:34:16.451031 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.451046 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.451058 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:34:16.451095 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.451097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.451113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.451810 (+ 697us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.451868 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.453066 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.453820 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.454359 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.454442 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.456694 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.456703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.456703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.456711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.458653 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.459042 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.459046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.459046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.459110 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.459334 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.459335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.459335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.459499 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.459583 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.459737 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.459822 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260928 06:34:16.460376 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8416fea2b34c42b14b34a9456eb996 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:33:56 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57978
I20260928 06:34:16.460889 8404 tablet_replica.cc:330] stopping tablet replica
I20260928 06:34:16.461009 8404 raft_consensus.cc:2229] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:16.461090 8404 raft_consensus.cc:2258] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:16.461431 8404 ts_tablet_manager.cc:1905] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:16.464011 8404 ts_tablet_manager.cc:1918] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:16.464102 8404 log.cc:1196] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/ff8416fea2b34c42b14b34a9456eb996
I20260928 06:34:16.464344 8404 ts_tablet_manager.cc:1939] T ff8416fea2b34c42b14b34a9456eb996 P 663f7e36c6574609bc20595718abb43f: Deleting consensus metadata
I20260928 06:34:16.464874 8056 catalog_manager.cc:4841] TS 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): tablet ff8416fea2b34c42b14b34a9456eb996 (table test-table [id=95661e5d32cd43d496579b755967ba36]) successfully deleted
W20260928 06:34:16.531487 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:16.531687 8401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:16.531713 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:16.532887 8401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:16.532932 8401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:16.532959 8401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:16.534475 8401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:41661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=36781
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:16.534727 8401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:16.534964 8401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:16.540006 8409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.540076 8401 server_base.cc:1034] running on GCE node
W20260928 06:34:16.539952 8412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:16.539958 8410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:16.540408 8401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:16.540669 8401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:16.541831 8401 hybrid_clock.cc:648] HybridClock initialized: now 1790577256541820 us; error 30 us; skew 500 ppm
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:16.544322 8401 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:34:16.544734 8401 webserver.cc:460] Webserver started at http://127.24.11.131:36781/ using document root <none> and password file <none>
I20260928 06:34:16.544950 8401 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:16.545020 8401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:16.546459 8401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:16.547039 8419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.547220 8401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:16.547338 8401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "9010f7b134564a7b839849bd6243eb38"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "f75b4cac2cbbee2e15687b278fc14d15"
server_key_iv: "d8e6b14d12448c9b3d3ff81a6c4bae3e"
server_key_version: "encryptionkey@0"
I20260928 06:34:16.547676 8401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:16.572927 8401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:16.575361 8401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:16.575485 8401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:16.575688 8401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:16.576021 8401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:16.576057 8401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.576100 8401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:16.576128 8401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.581995 8401 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:41661
I20260928 06:34:16.582075 8532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:41661 every 8 connection(s)
I20260928 06:34:16.582412 8401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:16.582626 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8401
Sep 28 06:34:16 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577256, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:16.589978 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.583250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.24.11.190:36965)
0928 06:34:16.583448 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.583450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.583492 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:34:16.583586 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.583587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.583598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.583676 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.584279 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.584788 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.585749 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.585953 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.586059 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.587358 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.587359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.587360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.587367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.588767 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.589133 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.589134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.589134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.589200 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.589398 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.589399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.589399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.589511 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.589724 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.589761 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.589871 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":182,"thread_start_us":60,"threads_started":1}
I20260928 06:34:16.592170 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.583250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:34037 (local address 127.24.11.190:36965)
0928 06:34:16.583314 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.583316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.583392 (+ 76us) server_negotiation.cc:407] Connection header received
0928 06:34:16.583923 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.583924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.583936 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.583981 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.584944 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.585562 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.586844 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.587043 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.587123 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.589293 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.589295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.589295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.589302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.590860 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.591255 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.591256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.591257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.591314 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.591556 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.591557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.591558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.591665 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.591732 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.592003 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.592079 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:34:16.592396 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.583084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:34037)
0928 06:34:16.583252 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.583295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.583825 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.584049 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.584053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.584625 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:16.584822 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.584827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.585647 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.585648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.586734 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.586736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.587132 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.587133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.587255 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.587732 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:16.587743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:16.589196 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:16.590940 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.590947 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.590950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.591169 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.591377 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:16.591377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:16.591378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:16.591493 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:16.591736 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:16.591743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:16.591878 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.591983 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.592143 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260928 06:34:16.592808 8533 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36965
I20260928 06:34:16.592909 8533 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:16.593081 8533 heartbeater.cc:511] Master 127.24.11.190:36965 requested a full tablet report, sending...
I20260928 06:34:16.593427 8070 ts_manager.cc:194] Registered new tserver with Master: 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661)
I20260928 06:34:16.593839 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:34037
I20260928 06:34:16.601377 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:16.604123 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.601785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.24.11.190:36965)
0928 06:34:16.601875 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.601877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.601972 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:34:16.602030 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.602031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.602064 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.602127 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:16.602302 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.602759 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.603515 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.603690 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.603738 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.603890 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:16.603933 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:34:16.604052 (+ 119us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33956: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 06:34:16.604185 8261 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33956: token expired
W20260928 06:34:16.604285 7998 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.11.190:36965)
I20260928 06:34:16.610136 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.604547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.24.11.190:36965)
0928 06:34:16.604586 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.604588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.604596 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:16.604618 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.604619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.604628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.604683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:16.605169 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.605686 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.606443 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.606638 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.606722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.607814 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:16.607816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:16.607816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:16.607823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:16.608993 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.609327 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.609328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.609328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.609383 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:16.609637 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:16.609637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:16.609638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:16.609756 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:16.609856 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.609964 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.610037 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:34:16.611176 8070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:16.612677 8070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:16.614862 8192 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb635492d7a4148ab62621a4a8ed293 (DEFAULT_TABLE table=test-table [id=47b0aed0c94640acae48dff4338a799d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:16.615097 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb635492d7a4148ab62621a4a8ed293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:16.616683 8150 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap starting.
I20260928 06:34:16.616806 8331 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb635492d7a4148ab62621a4a8ed293 (DEFAULT_TABLE table=test-table [id=47b0aed0c94640acae48dff4338a799d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:16.616952 8331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb635492d7a4148ab62621a4a8ed293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:16.617051 8150 tablet_bootstrap.cc:654] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:16.617703 8150 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: No bootstrap required, opened a new log
I20260928 06:34:16.617748 8150 ts_tablet_manager.cc:1397] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:16.617854 8150 raft_consensus.cc:348] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.617916 8150 raft_consensus.cc:374] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:16.617959 8150 raft_consensus.cc:729] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.618012 8150 consensus_queue.cc:260] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.618109 8150 ts_tablet_manager.cc:1428] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:16.618409 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.614531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41661 (local address 127.0.0.1:59818)
0928 06:34:16.614577 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.614588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.614625 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.614948 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.614949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.614951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:16.614993 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.614994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.616061 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.616062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.617124 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.617127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.618192 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.618193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.618273 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.618277 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.618308 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.618346 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:34:16.618482 8289 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Bootstrap starting.
I20260928 06:34:16.618813 8289 tablet_bootstrap.cc:654] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:16.618973 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.614620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.24.11.131:41661)
0928 06:34:16.614788 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.614793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.614803 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:16.614845 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.614847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.614860 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.614929 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:16.615081 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.615987 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.617259 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.618082 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.618666 (+ 584us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.618798 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.618831 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.618886 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260928 06:34:16.619412 8289 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: No bootstrap required, opened a new log
I20260928 06:34:16.619468 8289 ts_tablet_manager.cc:1397] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:16.619830 8289 raft_consensus.cc:348] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.619788 8467 tablet_service.cc:1467] Processing CreateTablet for tablet 1fb635492d7a4148ab62621a4a8ed293 (DEFAULT_TABLE table=test-table [id=47b0aed0c94640acae48dff4338a799d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:16.619907 8289 raft_consensus.cc:374] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:16.619948 8289 raft_consensus.cc:729] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663f7e36c6574609bc20595718abb43f, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.619997 8289 consensus_queue.cc:260] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.620092 8289 ts_tablet_manager.cc:1428] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:16.620126 8467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fb635492d7a4148ab62621a4a8ed293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:16.622970 8542 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Bootstrap starting.
I20260928 06:34:16.623625 8542 tablet_bootstrap.cc:654] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:16.624055 8542 log.cc:824] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:16.624675 8542 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: No bootstrap required, opened a new log
I20260928 06:34:16.624727 8542 ts_tablet_manager.cc:1397] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:16.626107 8542 raft_consensus.cc:348] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.626245 8542 raft_consensus.cc:374] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:16.626268 8542 raft_consensus.cc:729] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9010f7b134564a7b839849bd6243eb38, State: Initialized, Role: FOLLOWER
I20260928 06:34:16.626358 8542 consensus_queue.cc:260] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.626588 8542 ts_tablet_manager.cc:1428] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:16.626590 8533 heartbeater.cc:503] Master 127.24.11.190:36965 was elected leader, sending a full tablet report...
I20260928 06:34:16.645354 8264 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:16.645517 8264 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.645977 8264 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:34:16.650184 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.646295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58901 (local address 127.24.11.131:41661)
0928 06:34:16.646364 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.646367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.646377 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:16.646442 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.646443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.646454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.646532 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:16.647034 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.647745 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649191 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.649777 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649860 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.649870 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.650074 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.650115 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:34:16.650184 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.646393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:58557 (local address 127.24.11.130:44387)
0928 06:34:16.646439 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.646441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.646545 (+ 104us) server_negotiation.cc:407] Connection header received
0928 06:34:16.646582 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.646583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.646613 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.646687 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:16.646881 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.647627 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649194 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.649829 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649898 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.649969 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.650074 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.650117 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:34:16.650207 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.646202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41661 (local address 127.24.11.129:58901)
0928 06:34:16.646258 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.646270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.646334 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.646524 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.646526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.646528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:16.646956 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.646962 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.647823 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.647824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.649098 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.649101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.649853 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.649947 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.649953 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.650051 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.650101 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":44}
I20260928 06:34:16.650240 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.646313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.24.11.129:58557)
0928 06:34:16.646472 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.646480 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:16.646530 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:16.646688 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:16.646688 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:16.646690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:16.646757 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.646760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.647730 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.647731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.649098 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:16.649101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.649921 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.649921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:16.650001 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.650004 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:16.650054 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:34:16.650101 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":34,"thread_start_us":101,"threads_started":1}
I20260928 06:34:16.650645 8487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb635492d7a4148ab62621a4a8ed293" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9010f7b134564a7b839849bd6243eb38" is_pre_election: true
I20260928 06:34:16.650674 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb635492d7a4148ab62621a4a8ed293" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f7e36c6574609bc20595718abb43f" is_pre_election: true
I20260928 06:34:16.650835 8487 raft_consensus.cc:2454] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 0.
I20260928 06:34:16.650835 8351 raft_consensus.cc:2454] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 0.
I20260928 06:34:16.651069 8146 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, 9010f7b134564a7b839849bd6243eb38; no voters:
I20260928 06:34:16.651237 8264 raft_consensus.cc:2781] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:16.651316 8264 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:16.651335 8264 raft_consensus.cc:3037] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:16.652089 8264 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.652227 8264 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 election: Requested vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
I20260928 06:34:16.652405 8487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb635492d7a4148ab62621a4a8ed293" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9010f7b134564a7b839849bd6243eb38"
I20260928 06:34:16.652420 8351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fb635492d7a4148ab62621a4a8ed293" candidate_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663f7e36c6574609bc20595718abb43f"
I20260928 06:34:16.652487 8351 raft_consensus.cc:3037] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:16.652495 8487 raft_consensus.cc:3037] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:16.653388 8487 raft_consensus.cc:2454] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 1.
I20260928 06:34:16.653388 8351 raft_consensus.cc:2454] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfbb67e3e0742e1aaaa26a5ac2b19fe in term 1.
I20260928 06:34:16.653581 8146 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, 663f7e36c6574609bc20595718abb43f; no voters:
I20260928 06:34:16.653676 8264 raft_consensus.cc:2781] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:16.653829 8264 raft_consensus.cc:686] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 LEADER]: Becoming Leader. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Running, Role: LEADER
I20260928 06:34:16.653949 8264 consensus_queue.cc:237] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:16.654714 8069 catalog_manager.cc:5495] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129). New cstate: current_term: 1 leader_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } health_report { overall_health: UNKNOWN } } }
I20260928 06:34:16.665109 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.662584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.24.11.129:45907)
0928 06:34:16.662641 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:16.662643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:16.662651 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:16.662702 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:16.662703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:16.662718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:16.662782 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:16.662888 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.663438 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.664272 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:16.664465 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:16.664533 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:16.664689 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:16.664794 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:16.664849 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:16.664987 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:34:16.665021 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:34:16.666882 8351 raft_consensus.cc:1261] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Refusing update from remote peer 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:16.666914 8487 raft_consensus.cc:1261] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Refusing update from remote peer 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:16.667219 8264 consensus_queue.cc:1035] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:16.667394 8546 consensus_queue.cc:1035] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:16.669792 8549 mvcc.cc:205] Tried to move back new op lower bound from 7334204443304923136 to 7334204443255074816. Current Snapshot: MvccSnapshot[applied={T|T < 7334204443304923136}]
I20260928 06:34:16.670609 8551 mvcc.cc:205] Tried to move back new op lower bound from 7334204443304923136 to 7334204443255074816. Current Snapshot: MvccSnapshot[applied={T|T < 7334204443304923136 or (T in {7334204443304923136})}]
I20260928 06:34:16.671897 8070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:16.671980 8070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:16.672801 8070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Sending DeleteTablet for 3 replicas of tablet 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:16.673149 8467 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb635492d7a4148ab62621a4a8ed293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:16 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:59818
I20260928 06:34:16.673202 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8108
I20260928 06:34:16.673924 8331 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fb635492d7a4148ab62621a4a8ed293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:16 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:57978
I20260928 06:34:16.673991 8557 tablet_replica.cc:330] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: stopping tablet replica
I20260928 06:34:16.674023 8404 tablet_replica.cc:330] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: stopping tablet replica
I20260928 06:34:16.674068 8404 raft_consensus.cc:2229] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:16.674094 8557 raft_consensus.cc:2229] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:16.674122 8404 raft_consensus.cc:2258] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:16.674198 8557 raft_consensus.cc:2258] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:16.674616 8404 ts_tablet_manager.cc:1905] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:16.674685 8557 ts_tablet_manager.cc:1905] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:16.676995 8404 ts_tablet_manager.cc:1918] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:16.677026 8557 ts_tablet_manager.cc:1918] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:16.677076 8404 log.cc:1196] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:16.677116 8557 log.cc:1196] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:16.677349 8404 ts_tablet_manager.cc:1939] T 1fb635492d7a4148ab62621a4a8ed293 P 663f7e36c6574609bc20595718abb43f: Deleting consensus metadata
I20260928 06:34:16.677434 8557 ts_tablet_manager.cc:1939] T 1fb635492d7a4148ab62621a4a8ed293 P 9010f7b134564a7b839849bd6243eb38: Deleting consensus metadata
I20260928 06:34:16.677737 8056 catalog_manager.cc:4841] TS 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): tablet 1fb635492d7a4148ab62621a4a8ed293 (table test-table [id=47b0aed0c94640acae48dff4338a799d]) successfully deleted
I20260928 06:34:16.677896 8056 catalog_manager.cc:4841] TS 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): tablet 1fb635492d7a4148ab62621a4a8ed293 (table test-table [id=47b0aed0c94640acae48dff4338a799d]) successfully deleted
W20260928 06:34:16.680086 8056 connection.cc:535] client connection to 127.24.11.129:45907 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:34:16.680214 8056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:34:16.680322 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8263
I20260928 06:34:16.682412 8265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:16.682242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:45907 (local address 127.0.0.1:36152)
0928 06:34:16.682303 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:16.682364 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:34:16.682572 8056 catalog_manager.cc:4584] TS 4dfbb67e3e0742e1aaaa26a5ac2b19fe (127.24.11.129:45907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fb635492d7a4148ab62621a4a8ed293: Network error: Client connection negotiation failed: client connection to 127.24.11.129:45907: connect: Connection refused (error 111)
I20260928 06:34:16.685897 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8401
I20260928 06:34:16.691380 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8037
I20260928 06:34:36.696911 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--webserver_interface=127.24.11.190
--webserver_port=34387
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:36965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:36.780103 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:36.780402 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:36.780462 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:36.781801 8559 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:34:36.781888 8559 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:34:36.781910 8559 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:34:36.781930 8559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:36.781963 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:36.781996 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:36.782019 8559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:36.782044 8559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:36.783730 8559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:36965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=34387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:36.784071 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:36.784344 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:36.790052 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:36.789994 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:36.789994 8559 server_base.cc:1034] running on GCE node
W20260928 06:34:36.790014 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:36.790501 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:36.790784 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:36.791976 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790577276791963 us; error 31 us; skew 500 ppm
Sep 28 06:34:36 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:36.794808 8559 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:34:36.795312 8559 webserver.cc:460] Webserver started at http://127.24.11.190:34387/ using document root <none> and password file <none>
I20260928 06:34:36.795571 8559 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:36.795646 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:36.797117 8559 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:36.797930 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:36.798137 8559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:36.798251 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "307dba354c364493be9c978b003b7c90"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "ad879a04d6d94e5087fd735f224e12fc"
server_key_iv: "f4187483d3b8b5323fa482397d2aeee7"
server_key_version: "encryptionkey@0"
I20260928 06:34:36.798674 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:36.837247 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:36.839577 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:36.839756 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:36.844178 8559 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:36965
I20260928 06:34:36.844231 8626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:36965 every 8 connection(s)
I20260928 06:34:36.844658 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:36.846350 8627 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:34:36.847019 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap starting.
I20260928 06:34:36.852850 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8559
I20260928 06:34:36.853169 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:45907
--local_ip_for_outbound_sockets=127.24.11.129
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=33127
--webserver_interface=127.24.11.129
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:34:36.856334 8627 log.cc:824] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:36.858768 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:34:36.859098 8627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Bootstrap complete.
I20260928 06:34:36.861613 8627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:36.861883 8627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307dba354c364493be9c978b003b7c90, State: Initialized, Role: FOLLOWER
I20260928 06:34:36.862090 8627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:36.862186 8627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:36.862239 8627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:36.862287 8627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:34:36.863305 8627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:36.863426 8627 leader_election.cc:302] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307dba354c364493be9c978b003b7c90; no voters:
I20260928 06:34:36.863627 8627 leader_election.cc:288] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:34:36.863665 8631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:34:36.863788 8631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [term 3 LEADER]: Becoming Leader. State: Replica: 307dba354c364493be9c978b003b7c90, State: Running, Role: LEADER
I20260928 06:34:36.863878 8631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } }
I20260928 06:34:36.863929 8627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:36.864126 8633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307dba354c364493be9c978b003b7c90. Latest consensus state: current_term: 3 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:34:36.864336 8633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:36.864281 8632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "307dba354c364493be9c978b003b7c90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307dba354c364493be9c978b003b7c90" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36965 } } }
I20260928 06:34:36.864588 8632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:36.864741 8638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:36.865645 8638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:36.865773 8638 catalog_manager.cc:1259] Loaded cluster ID: 7991a185025543188ea00d5165a3432f
I20260928 06:34:36.865813 8638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:36.866899 8638 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:36.867091 8638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Loaded TSK: 0
I20260928 06:34:36.867154 8638 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 307dba354c364493be9c978b003b7c90: Loaded TSK: 1
F20260928 06:34:36.867375 8638 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7faf046049ad google::LogMessage::Fail() at ??:0
@ 0x7faf0460788a google::LogMessage::SendToLog() at ??:0
@ 0x7faf046044de google::LogMessage::Flush() at ??:0
@ 0x7faf04605a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7faf06d71104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7faf06d6ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7faf0c55e1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7faf0c55e48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7faf0c55e594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7faf0c524c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7faf0c52ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7faf064d20d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7faf064c7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7faf0732b6db start_thread at ??:0
@ 0x7faf0310261f clone at ??:0
W20260928 06:34:36.956861 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:36.957068 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:36.957093 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:36.958348 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:36.958398 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:36.958426 8629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:36.959897 8629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:45907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=33127
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:36.960139 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:36.960394 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:36.966714 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:36.966753 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:36.967060 8629 server_base.cc:1034] running on GCE node
W20260928 06:34:36.966874 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:36.967675 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:36.967942 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:36.969144 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790577276969127 us; error 33 us; skew 500 ppm
Sep 28 06:34:36 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:36.972020 8629 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:34:36.972481 8629 webserver.cc:460] Webserver started at http://127.24.11.129:33127/ using document root <none> and password file <none>
I20260928 06:34:36.972731 8629 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:36.972808 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:36.974345 8629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:36.975188 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:36.975399 8629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:36.975510 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "0492e01a443074ef31e216ca009e4253"
server_key_iv: "94af8bdf543822ccd47d4b1e79455535"
server_key_version: "encryptionkey@0"
I20260928 06:34:36.975925 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:36.996261 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:36.998945 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:36.999130 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:36.999387 8629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:36.999850 8685 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:34:37.001945 8629 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:34:37.002035 8629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:34:37.002086 8629 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:34:37.002838 8629 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:34:37.002898 8629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:37.002980 8685 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap starting.
I20260928 06:34:37.010829 8629 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:45907
I20260928 06:34:37.011021 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:45907 every 8 connection(s)
I20260928 06:34:37.011340 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:37.014746 8685 log.cc:824] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:37.017696 8685 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:34:37.018039 8685 tablet_bootstrap.cc:492] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Bootstrap complete.
I20260928 06:34:37.018838 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8629
I20260928 06:34:37.019120 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:44387
--local_ip_for_outbound_sockets=127.24.11.130
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=41209
--webserver_interface=127.24.11.130
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:34:37.019316 8685 ts_tablet_manager.cc:1397] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260928 06:34:37.020812 8685 raft_consensus.cc:348] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:37.021169 8685 raft_consensus.cc:729] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfbb67e3e0742e1aaaa26a5ac2b19fe, State: Initialized, Role: FOLLOWER
I20260928 06:34:37.021366 8685 consensus_queue.cc:260] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:37.021672 8685 ts_tablet_manager.cc:1428] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260928 06:34:37.069460 8795 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:34:37.012104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:49479)
0928 06:34:37.012291 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.012351 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:37.012884 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:37.069211 (+ 56327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260928 06:34:37.069768 8793 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:36965 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:36965: Client connection negotiation failed: client connection to 127.24.11.190:36965: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:34:37.070138 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.070008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:53649)
0928 06:34:37.070064 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.070108 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:34:37.070447 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.070355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:45781)
0928 06:34:37.070403 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.070427 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:34:37.070559 8793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:34:37.108210 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:37.108453 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:37.108493 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:37.109711 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:37.109776 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:37.109822 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:37.111299 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:44387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=41209
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:37.111557 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:37.111792 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:37.116925 8803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:37.116932 8804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:37.117033 8797 server_base.cc:1034] running on GCE node
W20260928 06:34:37.116998 8806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:37.117350 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:37.117623 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:37.118842 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790577277118823 us; error 53 us; skew 500 ppm
Sep 28 06:34:37 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577277, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:37.121284 8797 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:34:37.121790 8797 webserver.cc:460] Webserver started at http://127.24.11.130:41209/ using document root <none> and password file <none>
I20260928 06:34:37.122014 8797 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:37.122080 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:37.123440 8797 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:37.124046 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.124264 8797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:37.124354 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "663f7e36c6574609bc20595718abb43f"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "5eb78c8e2c3c15a3f47d8d6aa3a52c73"
server_key_iv: "c6e6f44fb848e75e450e8833003982fa"
server_key_version: "encryptionkey@0"
I20260928 06:34:37.124747 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:37.136408 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:37.138448 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:37.138612 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:37.138866 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:37.139170 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:37.139223 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.139263 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:37.139288 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.145746 8797 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:44387
I20260928 06:34:37.145807 8926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:44387 every 8 connection(s)
I20260928 06:34:37.146234 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:37.147310 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.146905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:51983)
0928 06:34:37.147081 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.147171 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260928 06:34:37.147748 8927 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:36965 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:36965: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
I20260928 06:34:37.148053 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.147965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:57435)
0928 06:34:37.148010 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.148036 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:34:37.148443 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.148371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:48079)
0928 06:34:37.148409 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.148429 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:34:37.148573 8927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:34:37.154753 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8797
I20260928 06:34:37.155084 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:41661
--local_ip_for_outbound_sockets=127.24.11.131
--tserver_master_addrs=127.24.11.190:36965
--webserver_port=36781
--webserver_interface=127.24.11.131
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:37.238477 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:37.238729 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:37.238771 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:37.239974 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:37.240046 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:37.240091 8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:37.241796 8930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:41661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=36781
--tserver_master_addrs=127.24.11.190:36965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:37.241995 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:37.242177 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:37.247241 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:37.247260 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:37.247478 8930 server_base.cc:1034] running on GCE node
W20260928 06:34:37.247241 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:37.247783 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:37.248056 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:37.249229 8930 hybrid_clock.cc:648] HybridClock initialized: now 1790577277249214 us; error 47 us; skew 500 ppm
Sep 28 06:34:37 dist-test-slave-38pg krb5kdc[7490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577277, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:37.251959 8930 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:34:37.252456 8930 webserver.cc:460] Webserver started at http://127.24.11.131:36781/ using document root <none> and password file <none>
I20260928 06:34:37.252683 8930 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:37.252729 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:37.254220 8930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:34:37.254953 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.255163 8930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:37.255259 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "9010f7b134564a7b839849bd6243eb38"
format_stamp: "Formatted at 2026-09-28 06:33:55 on dist-test-slave-38pg"
server_key: "f75b4cac2cbbee2e15687b278fc14d15"
server_key_iv: "d8e6b14d12448c9b3d3ff81a6c4bae3e"
server_key_version: "encryptionkey@0"
I20260928 06:34:37.255630 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:37.266419 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:37.268937 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:37.269093 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:37.269383 8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:37.269784 8930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:37.269842 8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.269879 8930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:37.269906 8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:37.276081 8930 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:41661
I20260928 06:34:37.276122 9058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:41661 every 8 connection(s)
I20260928 06:34:37.276487 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:37.277674 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.277164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:57543)
0928 06:34:37.277323 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.277447 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
W20260928 06:34:37.278185 9059 heartbeater.cc:644] Failed to heartbeat to 127.24.11.190:36965 (0 consecutive failures): Network error: Failed to ping master at 127.24.11.190:36965: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
I20260928 06:34:37.278507 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.278409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:50841)
0928 06:34:37.278460 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.278490 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:34:37.278797 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:37.278721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:56195)
0928 06:34:37.278761 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:37.278783 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:34:37.278916 9059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:34:37.280232 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 8930
I20260928 06:34:38.071241 9063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.070973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:38139)
0928 06:34:38.071138 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:38.071191 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260928 06:34:38.149124 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.148923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:46803)
0928 06:34:38.149037 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:38.149074 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260928 06:34:38.279500 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.279276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:44967)
0928 06:34:38.279405 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:38.279449 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 06:34:38.625005 9066 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:38.625144 9066 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:38.625447 9066 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
Sep 28 06:34:38 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.131@KRBTEST.COM
Sep 28 06:34:38 dist-test-slave-38pg krb5kdc[7490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577276, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.130@KRBTEST.COM
I20260928 06:34:38.637156 9067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.625675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41661 (local address 127.24.11.129:33685)
0928 06:34:38.625982 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:38.625993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:38.626063 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:38.627121 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:38.627125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:38.627643 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:38.627953 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:38.627963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.629118 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.629119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:38.630082 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:38.630086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.630698 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:38.630813 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:38.631507 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:38.631512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:38.633170 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:38.635503 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:38.635512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:38.635514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:38.635807 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:38.636081 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:38.636082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:38.636082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:38.636196 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:38.636603 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:38.636609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:38.636751 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:34:38.636861 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:34:38.637059 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":55,"thread_start_us":94,"threads_started":1}
I20260928 06:34:38.637173 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.625815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:33685 (local address 127.24.11.131:41661)
0928 06:34:38.625973 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:38.625985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:38.626119 (+ 134us) server_negotiation.cc:407] Connection header received
0928 06:34:38.626839 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:38.626841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:38.626883 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:38.627040 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:38.628105 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.629010 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630185 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.630591 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630737 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:38.633284 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:38.633288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:38.633288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:38.633300 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:38.635388 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:38.635897 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:38.635901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:38.635901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:38.636002 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:38.636279 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:38.636280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:38.636280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:38.636522 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:38.636612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:38.636914 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 06:34:38.637046 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260928 06:34:38.637217 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.626027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:44431 (local address 127.24.11.130:44387)
0928 06:34:38.626242 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:38.626252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:38.626280 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:34:38.626813 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:38.626817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:38.626883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:38.627041 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:38.628101 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.629215 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630179 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.630585 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630700 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:38.633719 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:38.633722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:38.633722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:38.633733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:38.635499 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:38.635935 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:38.635939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:38.635939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:38.636046 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:38.636266 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:38.636267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:38.636267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:38.636522 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:38.636612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:38.636930 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 06:34:38.637092 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260928 06:34:38.637274 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:38.625892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:44387 (local address 127.24.11.129:44431)
0928 06:34:38.625981 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:38.625993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:38.626051 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:38.627122 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:38.627125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:38.627643 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:38.627953 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:38.627963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.629321 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.629322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:38.630081 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:38.630084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:38.630684 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:38.630685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:38.630813 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:38.631453 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:38.631464 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:38.633605 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:38.635617 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:38.635619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:38.635619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:38.635847 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:38.636128 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:38.636129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:38.636129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:38.636199 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:38.636606 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:38.636612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:38.636752 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:34:38.636861 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:34:38.637140 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":144,"thread_start_us":35,"threads_started":1}
W20260928 06:34:38.638823 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
W20260928 06:34:38.638896 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:38.638913 8681 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe; no voters: 663f7e36c6574609bc20595718abb43f, 9010f7b134564a7b839849bd6243eb38
I20260928 06:34:38.639038 9066 raft_consensus.cc:2726] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:34:39.072029 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:39.071847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:60771)
0928 06:34:39.071937 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:39.071988 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 06:34:39.149896 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:39.149651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:37815)
0928 06:34:39.149804 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:39.149852 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260928 06:34:39.280237 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:39.280010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:34529)
0928 06:34:39.280135 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:39.280176 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260928 06:34:40.072876 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:40.072576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:38277)
0928 06:34:40.072735 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:40.072827 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260928 06:34:40.150820 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:40.150429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:48543)
0928 06:34:40.150697 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:40.150751 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":63,"threads_started":1}
I20260928 06:34:40.281011 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:40.280792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:59263)
0928 06:34:40.280928 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:40.280968 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 06:34:40.357266 9076 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:40.357369 9076 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:40.357623 9076 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
W20260928 06:34:40.358207 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
W20260928 06:34:40.358340 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:40.358377 8681 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe; no voters: 663f7e36c6574609bc20595718abb43f, 9010f7b134564a7b839849bd6243eb38
I20260928 06:34:40.358485 9076 raft_consensus.cc:2726] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:34:41.073657 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:41.073390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:49961)
0928 06:34:41.073572 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:41.073611 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260928 06:34:41.151556 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:41.151330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:51621)
0928 06:34:41.151469 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:41.151512 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260928 06:34:41.281837 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:41.281600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:39803)
0928 06:34:41.281744 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:41.281791 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260928 06:34:42.074512 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:42.074268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:51625)
0928 06:34:42.074416 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:42.074460 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260928 06:34:42.092166 9081 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:42.092288 9081 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:42.092481 9081 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
W20260928 06:34:42.092921 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
W20260928 06:34:42.092994 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:42.093045 8681 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe; no voters: 663f7e36c6574609bc20595718abb43f, 9010f7b134564a7b839849bd6243eb38
I20260928 06:34:42.093123 9081 raft_consensus.cc:2726] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:34:42.152354 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:42.152085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:32885)
0928 06:34:42.152245 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:42.152309 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260928 06:34:42.282516 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:42.282267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:45729)
0928 06:34:42.282427 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:42.282468 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260928 06:34:43.075357 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:43.075094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:44665)
0928 06:34:43.075264 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:43.075311 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260928 06:34:43.153079 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:43.152871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:57197)
0928 06:34:43.152993 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:43.153033 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 06:34:43.283351 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:43.283096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:37647)
0928 06:34:43.283260 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:43.283307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260928 06:34:44.076177 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:44.075916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:52399)
0928 06:34:44.076087 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:44.076133 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260928 06:34:44.153914 9088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:44.153668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:56547)
0928 06:34:44.153799 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:44.153867 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 06:34:44.284325 9089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:44.284031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:32879)
0928 06:34:44.284237 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:44.284278 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":106,"threads_started":1}
I20260928 06:34:44.741415 9090 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:44.741600 9090 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:44.741801 9090 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
W20260928 06:34:44.742173 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
W20260928 06:34:44.742244 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:44.742261 8681 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe; no voters: 663f7e36c6574609bc20595718abb43f, 9010f7b134564a7b839849bd6243eb38
I20260928 06:34:44.742329 9090 raft_consensus.cc:2726] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:34:45.077157 9091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:45.076882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:33237)
0928 06:34:45.077055 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:45.077106 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260928 06:34:45.154706 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:45.154445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:47275)
0928 06:34:45.154605 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:45.154647 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260928 06:34:45.285171 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:45.284908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:48151)
0928 06:34:45.285073 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:45.285121 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260928 06:34:46.078096 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:46.077848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:38147)
0928 06:34:46.078013 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:46.078050 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260928 06:34:46.155463 9095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:46.155216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:54147)
0928 06:34:46.155373 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:46.155415 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260928 06:34:46.285957 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:46.285731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:58353)
0928 06:34:46.285866 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:46.285909 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260928 06:34:47.078887 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:47.078652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:45407)
0928 06:34:47.078779 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:47.078840 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 06:34:47.156284 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:47.156019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:52321)
0928 06:34:47.156180 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:47.156236 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260928 06:34:47.286634 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:47.286397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:44237)
0928 06:34:47.286539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:47.286579 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260928 06:34:48.079658 9100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:48.079417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:37437)
0928 06:34:48.079563 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:48.079609 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 06:34:48.156960 9101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:48.156756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:41601)
0928 06:34:48.156879 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:48.156915 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260928 06:34:48.287551 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:48.287292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:42947)
0928 06:34:48.287456 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:48.287502 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260928 06:34:48.465039 9103 raft_consensus.cc:482] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:48.465137 9103 raft_consensus.cc:504] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9010f7b134564a7b839849bd6243eb38" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41661 } } peers { permanent_uuid: "4dfbb67e3e0742e1aaaa26a5ac2b19fe" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 45907 } } peers { permanent_uuid: "663f7e36c6574609bc20595718abb43f" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 44387 } }
I20260928 06:34:48.465364 9103 leader_election.cc:288] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661), 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387)
W20260928 06:34:48.465816 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663f7e36c6574609bc20595718abb43f (127.24.11.130:44387): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
W20260928 06:34:48.465883 8681 leader_election.cc:341] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9010f7b134564a7b839849bd6243eb38 (127.24.11.131:41661): Not found: Tablet not found: 1fb635492d7a4148ab62621a4a8ed293
I20260928 06:34:48.465899 8681 leader_election.cc:302] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dfbb67e3e0742e1aaaa26a5ac2b19fe; no voters: 663f7e36c6574609bc20595718abb43f, 9010f7b134564a7b839849bd6243eb38
I20260928 06:34:48.465974 9103 raft_consensus.cc:2726] T 1fb635492d7a4148ab62621a4a8ed293 P 4dfbb67e3e0742e1aaaa26a5ac2b19fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:34:49.080425 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:49.080197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:37677)
0928 06:34:49.080334 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:49.080378 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260928 06:34:49.157862 9105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:49.157521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:50519)
0928 06:34:49.157774 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:49.157815 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":68,"threads_started":1}
I20260928 06:34:49.288411 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:49.288144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:49979)
0928 06:34:49.288314 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:49.288362 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260928 06:34:50.081281 9107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:50.081026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:34431)
0928 06:34:50.081185 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:50.081231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260928 06:34:50.158651 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:50.158420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:45865)
0928 06:34:50.158561 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:50.158605 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260928 06:34:50.289222 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:50.288984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:51725)
0928 06:34:50.289133 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:50.289173 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260928 06:34:51.082028 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:51.081801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:36451)
0928 06:34:51.081936 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:51.081980 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260928 06:34:51.159358 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:51.159091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:37895)
0928 06:34:51.159238 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:51.159280 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260928 06:34:51.289994 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:51.289786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:56057)
0928 06:34:51.289908 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:51.289948 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 06:34:52.082919 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:52.082696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.129:44913)
0928 06:34:52.082834 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:52.082875 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 06:34:52.160118 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:52.159912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.130:33307)
0928 06:34:52.160034 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:52.160074 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 06:34:52.287511 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8629
I20260928 06:34:52.290800 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:52.290540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36965 (local address 127.24.11.131:59633)
0928 06:34:52.290718 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:52.290756 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.190:36965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260928 06:34:52.292838 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8797
I20260928 06:34:52.297338 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 8930
2026-09-28T06:34:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58170 ms)
[----------] 30 tests from SecurityITest (236946 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:34:52 dist-test-slave-38pg krb5kdc[9118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:34:52 dist-test-slave-38pg krb5kdc[9118](info): set up 2 sockets
Sep 28 06:34:52 dist-test-slave-38pg krb5kdc[9118](info): commencing operation
krb5kdc: starting...
W20260928 06:34:53.374466 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:34:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:34:53.497290 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36301
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:53.580667 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:53.580950 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:53.580991 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:53.582180 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:53.582257 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:53.582283 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:53.582305 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:53.582335 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:53.583838 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:36301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:53.584116 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:53.584355 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:53.589233 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:53.589249 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:53.589241 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:53.589687 9131 server_base.cc:1034] running on GCE node
I20260928 06:34:53.589970 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:53.590305 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:53.591486 9131 hybrid_clock.cc:648] HybridClock initialized: now 1790577293591467 us; error 50 us; skew 500 ppm
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:53.601859 9131 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:34:53.602414 9131 webserver.cc:460] Webserver started at http://127.24.11.190:32877/ using document root <none> and password file <none>
I20260928 06:34:53.602655 9131 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:53.602708 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:53.602833 9131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:53.603945 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "4b1e158d5ece4e799b70669c36d225e3"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "3255102ff5e69f41f29fdcad7bcda6d7"
server_key_iv: "06be3232dba447a933df8468917e2748"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.604326 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "4b1e158d5ece4e799b70669c36d225e3"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "3255102ff5e69f41f29fdcad7bcda6d7"
server_key_iv: "06be3232dba447a933df8468917e2748"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.605792 9131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:34:53.606655 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:53.606868 9131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:53.606980 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "4b1e158d5ece4e799b70669c36d225e3"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "3255102ff5e69f41f29fdcad7bcda6d7"
server_key_iv: "06be3232dba447a933df8468917e2748"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.607059 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:53.630957 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:53.633296 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:53.633482 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:53.637761 9131 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:36301
I20260928 06:34:53.637795 9198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:36301 every 8 connection(s)
I20260928 06:34:53.638152 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:53.638967 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:53.641263 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: Bootstrap starting.
I20260928 06:34:53.641917 9199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:53.642203 9199 log.cc:824] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:53.642778 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9131
I20260928 06:34:53.642861 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:34:53.643008 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: No bootstrap required, opened a new log
I20260928 06:34:53.643065 24622 external_mini_cluster.cc:1428] Setting key 187f3a05dfccb56bd8b5f68751e78cfd
I20260928 06:34:53.644817 9199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } }
I20260928 06:34:53.644974 9199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:53.645041 9199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b1e158d5ece4e799b70669c36d225e3, State: Initialized, Role: FOLLOWER
I20260928 06:34:53.645195 9199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } }
I20260928 06:34:53.645260 9199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:53.645282 9199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:53.645329 9199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:53.646296 9199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } }
I20260928 06:34:53.646417 9199 leader_election.cc:302] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b1e158d5ece4e799b70669c36d225e3; no voters:
I20260928 06:34:53.646611 9199 leader_election.cc:288] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:53.646633 9204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:53.646745 9204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b1e158d5ece4e799b70669c36d225e3, State: Running, Role: LEADER
I20260928 06:34:53.646888 9199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:53.646894 9204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } }
I20260928 06:34:53.647405 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b1e158d5ece4e799b70669c36d225e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } } }
I20260928 06:34:53.647492 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:53.647439 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b1e158d5ece4e799b70669c36d225e3. Latest consensus state: current_term: 1 leader_uuid: "4b1e158d5ece4e799b70669c36d225e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b1e158d5ece4e799b70669c36d225e3" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36301 } } }
I20260928 06:34:53.647545 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:53.648443 9212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:53.648934 9212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:53.650604 9212 catalog_manager.cc:1347] Generated new cluster ID: c570c55ce3714a7e8b2112a36ca44c53
I20260928 06:34:53.650658 9212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:53.655376 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:53.643627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.24.11.190:36301)
0928 06:34:53.643845 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:53.643854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:53.643890 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:34:53.644838 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:53.644847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:53.644901 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:53.645069 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:53.645860 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.647027 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.648117 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.648724 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.648866 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:53.651583 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:53.651587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:53.651587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:53.651599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:53.653353 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:53.653803 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:53.653806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:53.653807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:53.653922 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:53.654144 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:53.654145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:53.654145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:53.654380 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:53.655072 (+ 692us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:53.655128 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:34:53.655234 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260928 06:34:53.680843 9212 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:53.681659 9212 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:53.685235 9212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: Generated new TSK 0
I20260928 06:34:53.685422 9212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:34:53.717078 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:53.805269 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:53.805485 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:53.805510 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:53.806736 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:53.806782 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:53.806808 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:53.808393 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:53.808652 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:53.808910 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:53.814312 9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:53.814352 9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:53.814352 9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:53.814459 9225 server_base.cc:1034] running on GCE node
I20260928 06:34:53.814762 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:53.815034 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:53.816195 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790577293816186 us; error 29 us; skew 500 ppm
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:53.818709 9225 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:34:53.819136 9225 webserver.cc:460] Webserver started at http://127.24.11.129:44941/ using document root <none> and password file <none>
I20260928 06:34:53.819361 9225 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:53.819428 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:53.819581 9225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:53.820677 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "dfe0e363341614808df8f6b6f89e627d"
server_key_iv: "b742b4f94e88dc8be75381d5766ebd5f"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.821039 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "dfe0e363341614808df8f6b6f89e627d"
server_key_iv: "b742b4f94e88dc8be75381d5766ebd5f"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.822472 9225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:53.823249 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:53.823463 9225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:53.823573 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "dfe0e363341614808df8f6b6f89e627d"
server_key_iv: "b742b4f94e88dc8be75381d5766ebd5f"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.823648 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:53.840930 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:53.843186 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:53.843381 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:53.843627 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:53.843937 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:53.843993 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:53.844038 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:53.844061 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:53.850512 9225 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:37027
I20260928 06:34:53.850562 9353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:37027 every 8 connection(s)
I20260928 06:34:53.850898 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:53.852824 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9225
I20260928 06:34:53.852918 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:53.853118 24622 external_mini_cluster.cc:1428] Setting key f5cac9491e3c3eaaa7d2dc9cd2b44857
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:53.861407 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:53.851665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:51101 (local address 127.24.11.190:36301)
0928 06:34:53.851748 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:53.851750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:53.851947 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:34:53.852424 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:53.852425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:53.852439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:53.852494 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:53.853774 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.854272 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.855343 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.855576 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.855655 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:53.857959 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:53.857962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:53.857962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:53.857976 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:53.859822 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:53.860304 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:53.860305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:53.860306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:53.860369 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:53.860681 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:53.860682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:53.860683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:53.860854 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:53.860931 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:53.861210 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 06:34:53.861307 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:34:53.861531 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:53.851620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36301 (local address 127.24.11.129:51101)
0928 06:34:53.851808 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:53.851847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:53.852329 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:53.852571 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:53.852575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:53.853355 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:53.853679 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:53.853687 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.854383 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.854384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:53.855238 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:53.855241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:53.855665 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:53.855666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:53.855818 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:53.856404 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:53.856414 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:53.857858 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:53.859922 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:53.859930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:53.859932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:53.860216 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:53.860464 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:53.860465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:53.860465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:53.860598 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:53.860928 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:53.860934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:53.861065 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:34:53.861154 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:34:53.861305 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
I20260928 06:34:53.862136 9354 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36301
I20260928 06:34:53.862401 9354 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:53.862660 9354 heartbeater.cc:511] Master 127.24.11.190:36301 requested a full tablet report, sending...
I20260928 06:34:53.863214 9163 ts_manager.cc:194] Registered new tserver with Master: 14a85291fc4a4f0bbce4fb197cd1bc0f (127.24.11.129:37027)
I20260928 06:34:53.863850 9163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:51101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:34:53.880206 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:53.964395 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:53.964653 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:53.964692 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:53.965952 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:53.966018 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:53.966063 9359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:53.967605 9359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:53.967900 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:53.968140 9359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:53.973440 9367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:53.973451 9365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:53.973482 9359 server_base.cc:1034] running on GCE node
W20260928 06:34:53.973451 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:53.973984 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:53.974236 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:53.975378 9359 hybrid_clock.cc:648] HybridClock initialized: now 1790577293975367 us; error 31 us; skew 500 ppm
Sep 28 06:34:53 dist-test-slave-38pg krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:53.978065 9359 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:34:53.978570 9359 webserver.cc:460] Webserver started at http://127.24.11.130:42803/ using document root <none> and password file <none>
I20260928 06:34:53.978825 9359 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:53.978875 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:53.979004 9359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:53.980120 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "96b5f5e7b7844230b517c9407db523cc"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "e9fb16f736717a1acba082c4820784d1"
server_key_iv: "fff802ed860ff8d566a9eec5a901d192"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.980506 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "96b5f5e7b7844230b517c9407db523cc"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "e9fb16f736717a1acba082c4820784d1"
server_key_iv: "fff802ed860ff8d566a9eec5a901d192"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.982002 9359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:53.982838 9374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:53.983021 9359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:34:53.983134 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "96b5f5e7b7844230b517c9407db523cc"
format_stamp: "Formatted at 2026-09-28 06:34:53 on dist-test-slave-38pg"
server_key: "e9fb16f736717a1acba082c4820784d1"
server_key_iv: "fff802ed860ff8d566a9eec5a901d192"
server_key_version: "encryptionkey@0"
I20260928 06:34:53.983211 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.014894 9359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.017496 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.017733 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.017987 9359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:54.018327 9359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:54.018385 9359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.018451 9359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:54.018498 9359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.024807 9359 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:39713
I20260928 06:34:54.024858 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:39713 every 8 connection(s)
I20260928 06:34:54.025233 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:54.025578 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9359
I20260928 06:34:54.025712 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:54.025957 24622 external_mini_cluster.cc:1428] Setting key c3d13cdd1c5b5030e18aa8eea82daefb
Sep 28 06:34:54 dist-test-slave-38pg krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577293, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
I20260928 06:34:54.036741 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.026059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:33459 (local address 127.24.11.190:36301)
0928 06:34:54.026124 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.026127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.026362 (+ 235us) server_negotiation.cc:407] Connection header received
0928 06:34:54.027198 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.027199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.027209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.027285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.028706 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.029236 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.030892 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.031088 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.031155 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.033750 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.033752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.033752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:54.033760 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.035317 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.035735 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.035737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.035738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.035797 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.036059 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.036060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.036060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.036180 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.036264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.036540 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.036636 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:34:54.036921 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.026016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36301 (local address 127.24.11.130:33459)
0928 06:34:54.026196 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.026253 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.027044 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.027376 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.027380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.028273 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:54.028590 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.028597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.029348 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.029349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.030794 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.030798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.031183 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.031185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.031343 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.031902 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:54.031913 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:54.033651 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:54.035412 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:54.035420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:54.035422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:54.035650 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:54.035868 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:54.035868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:54.035869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:54.035989 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:54.036255 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:54.036262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:54.036408 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.036520 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.036680 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260928 06:34:54.037449 9488 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36301
I20260928 06:34:54.037581 9488 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
I20260928 06:34:54.037809 9488 heartbeater.cc:511] Master 127.24.11.190:36301 requested a full tablet report, sending...
I20260928 06:34:54.038302 9163 ts_manager.cc:194] Registered new tserver with Master: 96b5f5e7b7844230b517c9407db523cc (127.24.11.130:39713)
I20260928 06:34:54.038688 9163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:33459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:34:54.053072 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:54.134868 9493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:54.135089 9493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:54.135113 9493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:54.136243 9493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:54.136277 9493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:54.136304 9493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:54.137782 9493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:36301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:54.137969 9493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:54.138141 9493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:54.143234 9499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:54.143294 9501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.143499 9493 server_base.cc:1034] running on GCE node
W20260928 06:34:54.143294 9498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.143815 9493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:54.144081 9493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:54.145226 9493 hybrid_clock.cc:648] HybridClock initialized: now 1790577294145216 us; error 28 us; skew 500 ppm
Sep 28 06:34:54 dist-test-slave-38pg krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577294, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:54.148062 9493 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:34:54.148677 9493 webserver.cc:460] Webserver started at http://127.24.11.131:36597/ using document root <none> and password file <none>
I20260928 06:34:54.148919 9493 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:54.148989 9493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:54.149130 9493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:54.150282 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "5ee5b0a7b0d545ee8beabce75ef31236"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "d02b0a08556d9ce2462996ad442d584f"
server_key_iv: "898a700e81136e7bdc490ae137747a22"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.150660 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee5b0a7b0d545ee8beabce75ef31236"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "d02b0a08556d9ce2462996ad442d584f"
server_key_iv: "898a700e81136e7bdc490ae137747a22"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.152020 9493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:54.152830 9508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.153132 9493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:54.153250 9493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "5ee5b0a7b0d545ee8beabce75ef31236"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "d02b0a08556d9ce2462996ad442d584f"
server_key_iv: "898a700e81136e7bdc490ae137747a22"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.153326 9493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.189870 9493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.191964 9493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.192139 9493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.192392 9493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:54.192729 9493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:54.192785 9493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.192819 9493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:54.192862 9493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.198622 9493 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:40271
I20260928 06:34:54.198668 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:40271 every 8 connection(s)
I20260928 06:34:54.198989 9493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:34:54 dist-test-slave-38pg krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577294, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:54.208456 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.199688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40027 (local address 127.24.11.190:36301)
0928 06:34:54.199767 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.199769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.199986 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:34:54.200477 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.200478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.200489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.200554 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.201708 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.202234 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.203284 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.203471 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.203527 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.205672 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.205674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.205674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:54.205681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.207184 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.207587 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.207588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.207588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.207656 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.207921 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.207922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.207922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.208032 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.208086 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.208322 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.208385 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:34:54.208617 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.199606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:36301 (local address 127.24.11.131:40027)
0928 06:34:54.199825 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.199869 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.200376 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.200724 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.200729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.201334 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:54.201606 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.201612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.202324 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.202325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.203205 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.203208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.203545 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.203545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.203644 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.204137 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:54.204145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:54.205523 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:54.207265 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:54.207271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:54.207272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:54.207490 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:54.207721 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:54.207722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:54.207722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:54.207859 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:54.208088 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:54.208092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:54.208205 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.208293 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.208415 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260928 06:34:54.208745 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9493
I20260928 06:34:54.208863 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:54.209065 24622 external_mini_cluster.cc:1428] Setting key fa0120227f47b6c86c03bc876e077265
I20260928 06:34:54.209081 9622 heartbeater.cc:347] Connected to a master server at 127.24.11.190:36301
I20260928 06:34:54.209172 9622 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:54.209398 9622 heartbeater.cc:511] Master 127.24.11.190:36301 requested a full tablet report, sending...
I20260928 06:34:54.209846 9162 ts_manager.cc:194] Registered new tserver with Master: 5ee5b0a7b0d545ee8beabce75ef31236 (127.24.11.131:40271)
I20260928 06:34:54.210250 9162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:40027
I20260928 06:34:54.221325 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:54.229110 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.222762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41444 (local address 127.24.11.190:36301)
0928 06:34:54.222851 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.222853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.222907 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:34:54.222951 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.222952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.222961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.223009 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.223570 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.224060 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.224754 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.224931 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.224980 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.226321 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.226323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.226323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:54.226330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.227876 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.228217 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.228218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.228218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.228288 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:54.228461 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:54.228462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:54.228462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:54.228605 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.228681 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.228878 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.228950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:34:54.231019 9162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:54.232694 9162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:54.240020 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:37027 (local address 127.0.0.1:38208)
0928 06:34:54.236116 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.236126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.236208 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.236503 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.236504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.236506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.236539 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.236540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.237438 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.237439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.238712 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.238714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.239709 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.239793 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.239809 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.239885 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.239939 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260928 06:34:54.240100 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.24.11.129:37027)
0928 06:34:54.236064 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.236068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.236180 (+ 112us) server_negotiation.cc:407] Connection header received
0928 06:34:54.236321 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.236324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.236341 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.236397 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.236631 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.237327 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.238805 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.239610 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239697 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.239762 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.239945 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.239996 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":109,"threads_started":1}
I20260928 06:34:54.240198 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39713 (local address 127.0.0.1:53912)
0928 06:34:54.235681 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.235782 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.235868 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.236253 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.236256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.236259 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.236345 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.236347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.237935 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.237936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.238937 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.238939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.239944 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.240028 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.240032 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.240067 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.240113 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":55,"threads_started":1}
I20260928 06:34:54.240553 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53912 (local address 127.24.11.130:39713)
0928 06:34:54.236043 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.236048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.236061 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:34:54.236104 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.236107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.236125 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.236435 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.236508 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.237821 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239023 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.239819 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.240272 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.240366 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.240394 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.240446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":123,"threads_started":1}
I20260928 06:34:54.240929 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:40271 (local address 127.0.0.1:53330)
0928 06:34:54.236131 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.236143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.236222 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.236441 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.236442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.236444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.236502 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.236504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.239007 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.239687 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.239689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.240684 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.240685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.240745 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.240748 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.240814 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.240862 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":82,"threads_started":1}
I20260928 06:34:54.240983 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.235831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.24.11.131:40271)
0928 06:34:54.236016 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.236021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.236292 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:34:54.236340 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.236343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.236360 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.236442 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.236564 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.237822 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.239789 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.240521 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.240583 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.240646 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.240854 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.240902 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
I20260928 06:34:54.241379 9422 tablet_service.cc:1467] Processing CreateTablet for tablet 35f324c1a5344155a0e102d9d554d966 (DEFAULT_TABLE table=test-table [id=9fd45fa4f1494526aa234d52ce882e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:54.241469 9288 tablet_service.cc:1467] Processing CreateTablet for tablet 35f324c1a5344155a0e102d9d554d966 (DEFAULT_TABLE table=test-table [id=9fd45fa4f1494526aa234d52ce882e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:54.241797 9422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f324c1a5344155a0e102d9d554d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:54.241835 9288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f324c1a5344155a0e102d9d554d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:54.241811 9556 tablet_service.cc:1467] Processing CreateTablet for tablet 35f324c1a5344155a0e102d9d554d966 (DEFAULT_TABLE table=test-table [id=9fd45fa4f1494526aa234d52ce882e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:54.242133 9556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35f324c1a5344155a0e102d9d554d966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:54.244397 9639 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Bootstrap starting.
I20260928 06:34:54.245072 9639 tablet_bootstrap.cc:654] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:54.245452 9640 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Bootstrap starting.
I20260928 06:34:54.245932 9639 log.cc:824] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:54.246233 9640 tablet_bootstrap.cc:654] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:54.246289 9641 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Bootstrap starting.
I20260928 06:34:54.246791 9639 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: No bootstrap required, opened a new log
I20260928 06:34:54.246913 9639 ts_tablet_manager.cc:1397] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:34:54.246937 9640 log.cc:824] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:54.247087 9641 tablet_bootstrap.cc:654] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:54.247670 9641 log.cc:824] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:54.247812 9640 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: No bootstrap required, opened a new log
I20260928 06:34:54.247893 9640 ts_tablet_manager.cc:1397] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:34:54.248478 9641 tablet_bootstrap.cc:492] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: No bootstrap required, opened a new log
I20260928 06:34:54.248541 9641 ts_tablet_manager.cc:1397] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:54.249281 9639 raft_consensus.cc:348] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.249488 9639 raft_consensus.cc:374] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:54.249596 9639 raft_consensus.cc:729] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14a85291fc4a4f0bbce4fb197cd1bc0f, State: Initialized, Role: FOLLOWER
I20260928 06:34:54.249747 9639 consensus_queue.cc:260] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.250056 9639 ts_tablet_manager.cc:1428] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:54.250084 9354 heartbeater.cc:503] Master 127.24.11.190:36301 was elected leader, sending a full tablet report...
I20260928 06:34:54.250164 9641 raft_consensus.cc:348] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.250166 9640 raft_consensus.cc:348] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.250320 9641 raft_consensus.cc:374] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:54.250320 9640 raft_consensus.cc:374] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:54.250360 9641 raft_consensus.cc:729] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee5b0a7b0d545ee8beabce75ef31236, State: Initialized, Role: FOLLOWER
I20260928 06:34:54.250383 9640 raft_consensus.cc:729] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b5f5e7b7844230b517c9407db523cc, State: Initialized, Role: FOLLOWER
I20260928 06:34:54.250492 9641 consensus_queue.cc:260] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.250648 9640 consensus_queue.cc:260] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.250743 9641 ts_tablet_manager.cc:1428] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:34:54.250903 9622 heartbeater.cc:503] Master 127.24.11.190:36301 was elected leader, sending a full tablet report...
I20260928 06:34:54.250965 9640 ts_tablet_manager.cc:1428] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:54.251083 9488 heartbeater.cc:503] Master 127.24.11.190:36301 was elected leader, sending a full tablet report...
W20260928 06:34:54.275996 9489 tablet.cc:2369] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:54.324537 9646 raft_consensus.cc:482] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:54.324668 9646 raft_consensus.cc:504] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.325150 9646 leader_election.cc:288] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14a85291fc4a4f0bbce4fb197cd1bc0f (127.24.11.129:37027), 96b5f5e7b7844230b517c9407db523cc (127.24.11.130:39713)
I20260928 06:34:54.328152 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.325370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:37027 (local address 127.24.11.131:38991)
0928 06:34:54.325452 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.325464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.325514 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.325711 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.325712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.325714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.325788 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.325790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.326505 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.326506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.327344 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.327346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.327879 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.327880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.327966 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.327969 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.328038 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.328079 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 06:34:54.328151 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.325446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:38991 (local address 127.24.11.129:37027)
0928 06:34:54.325495 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.325497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.325508 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:34:54.325615 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.325615 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.325626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.325703 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.325875 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.326417 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.327430 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.327794 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.327883 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.327891 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.328045 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.328080 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:34:54.328327 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.325387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39713 (local address 127.24.11.131:34179)
0928 06:34:54.325584 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.325593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.325675 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.325836 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.325837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.325840 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.325918 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.325919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.326649 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.326651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.327460 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.327462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.328056 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.328056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.328130 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.328133 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.328209 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.328249 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":125,"threads_started":1}
I20260928 06:34:54.328316 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.325443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:34179 (local address 127.24.11.130:39713)
0928 06:34:54.325487 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.325488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.325664 (+ 176us) server_negotiation.cc:407] Connection header received
0928 06:34:54.325730 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.325732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.325744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.325810 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.325997 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.326562 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.327548 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.327965 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.328056 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.328064 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.328218 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.328256 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:34:54.328725 9442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f324c1a5344155a0e102d9d554d966" candidate_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b5f5e7b7844230b517c9407db523cc" is_pre_election: true
I20260928 06:34:54.328712 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f324c1a5344155a0e102d9d554d966" candidate_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" is_pre_election: true
I20260928 06:34:54.328912 9442 raft_consensus.cc:2454] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee5b0a7b0d545ee8beabce75ef31236 in term 0.
I20260928 06:34:54.328922 9308 raft_consensus.cc:2454] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee5b0a7b0d545ee8beabce75ef31236 in term 0.
I20260928 06:34:54.329147 9509 leader_election.cc:302] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee5b0a7b0d545ee8beabce75ef31236, 96b5f5e7b7844230b517c9407db523cc; no voters:
I20260928 06:34:54.329321 9646 raft_consensus.cc:2781] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:54.329371 9646 raft_consensus.cc:482] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:54.329387 9646 raft_consensus.cc:3037] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:54.330140 9646 raft_consensus.cc:504] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.330289 9646 leader_election.cc:288] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [CANDIDATE]: Term 1 election: Requested vote from peers 14a85291fc4a4f0bbce4fb197cd1bc0f (127.24.11.129:37027), 96b5f5e7b7844230b517c9407db523cc (127.24.11.130:39713)
I20260928 06:34:54.330457 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f324c1a5344155a0e102d9d554d966" candidate_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f"
I20260928 06:34:54.330457 9442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35f324c1a5344155a0e102d9d554d966" candidate_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96b5f5e7b7844230b517c9407db523cc"
I20260928 06:34:54.330544 9308 raft_consensus.cc:3037] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:54.330544 9442 raft_consensus.cc:3037] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:54.331475 9308 raft_consensus.cc:2454] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee5b0a7b0d545ee8beabce75ef31236 in term 1.
I20260928 06:34:54.331475 9442 raft_consensus.cc:2454] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee5b0a7b0d545ee8beabce75ef31236 in term 1.
I20260928 06:34:54.331655 9509 leader_election.cc:302] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ee5b0a7b0d545ee8beabce75ef31236, 96b5f5e7b7844230b517c9407db523cc; no voters:
I20260928 06:34:54.331756 9646 raft_consensus.cc:2781] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:54.331905 9646 raft_consensus.cc:686] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee5b0a7b0d545ee8beabce75ef31236, State: Running, Role: LEADER
I20260928 06:34:54.332024 9646 consensus_queue.cc:237] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } }
I20260928 06:34:54.332916 9163 catalog_manager.cc:5495] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee5b0a7b0d545ee8beabce75ef31236 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee5b0a7b0d545ee8beabce75ef31236" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 40271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 } health_report { overall_health: UNKNOWN } } }
W20260928 06:34:54.351637 9355 tablet.cc:2369] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:54.360772 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.358343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.24.11.131:40271)
0928 06:34:54.358415 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.358417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.358426 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:54.358465 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.358466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.358492 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.358571 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:54.358700 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.359182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.359957 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.360170 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.360243 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.360412 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:54.360524 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:54.360623 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.360641 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.360662 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:34:54.362838 9442 raft_consensus.cc:1261] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 1 FOLLOWER]: Refusing update from remote peer 5ee5b0a7b0d545ee8beabce75ef31236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:54.362861 9308 raft_consensus.cc:1261] T 35f324c1a5344155a0e102d9d554d966 P 14a85291fc4a4f0bbce4fb197cd1bc0f [term 1 FOLLOWER]: Refusing update from remote peer 5ee5b0a7b0d545ee8beabce75ef31236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:54.363214 9649 consensus_queue.cc:1035] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96b5f5e7b7844230b517c9407db523cc" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 39713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:54.363341 9646 consensus_queue.cc:1035] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14a85291fc4a4f0bbce4fb197cd1bc0f" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:54.365837 9652 mvcc.cc:205] Tried to move back new op lower bound from 7334204597707448320 to 7334204597584838656. Current Snapshot: MvccSnapshot[applied={T|T < 7334204597707448320}]
I20260928 06:34:54.366150 9654 mvcc.cc:205] Tried to move back new op lower bound from 7334204597707448320 to 7334204597584838656. Current Snapshot: MvccSnapshot[applied={T|T < 7334204597707448320}]
I20260928 06:34:54.368458 9163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:54.368544 9163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:54.369285 9163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b1e158d5ece4e799b70669c36d225e3: Sending DeleteTablet for 3 replicas of tablet 35f324c1a5344155a0e102d9d554d966
I20260928 06:34:54.369652 9288 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f324c1a5344155a0e102d9d554d966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:54 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:38208
I20260928 06:34:54.369660 9422 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f324c1a5344155a0e102d9d554d966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:54 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:53912
I20260928 06:34:54.369940 9556 tablet_service.cc:1514] Processing DeleteTablet for tablet 35f324c1a5344155a0e102d9d554d966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:54 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:53330
I20260928 06:34:54.369971 9662 tablet_replica.cc:330] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc: stopping tablet replica
I20260928 06:34:54.370098 9662 raft_consensus.cc:2229] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:54.370208 9662 raft_consensus.cc:2258] T 35f324c1a5344155a0e102d9d554d966 P 96b5f5e7b7844230b517c9407db523cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:54.370229 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9225
W20260928 06:34:54.375272 9148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:37027 (error 108)
I20260928 06:34:54.375628 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9359
I20260928 06:34:54.375892 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.375748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:37027 (local address 127.0.0.1:38220)
0928 06:34:54.375814 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.375852 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 06:34:54.375999 9148 catalog_manager.cc:4584] TS 14a85291fc4a4f0bbce4fb197cd1bc0f (127.24.11.129:37027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35f324c1a5344155a0e102d9d554d966: Network error: Client connection negotiation failed: client connection to 127.24.11.129:37027: connect: Connection refused (error 111)
I20260928 06:34:54.377672 9664 tablet_replica.cc:330] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: stopping tablet replica
I20260928 06:34:54.377810 9664 raft_consensus.cc:2229] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:34:54.377931 9664 raft_consensus.cc:2258] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:54.381268 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.381137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:39713 (local address 127.0.0.1:53924)
0928 06:34:54.381201 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.381234 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:39713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:34:54.381268 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9493
I20260928 06:34:54.381438 9664 ts_tablet_manager.cc:1905] T 35f324c1a5344155a0e102d9d554d966 P 5ee5b0a7b0d545ee8beabce75ef31236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:54.386296 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9131
I20260928 06:34:54.386439 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.386282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:40271 (local address 127.0.0.1:53342)
0928 06:34:54.386357 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.386395 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.131:40271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
2026-09-28T06:34:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2089 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:54Z Disabled control of system clock
I20260928 06:34:54.415103 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38205
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:38205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:34:54.494635 9669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:54.494886 9669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:54.494908 9669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:54.496030 9669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:54.496070 9669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:54.496083 9669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:54.496096 9669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:54.496109 9669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:54.497407 9669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:38205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:54.497673 9669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:54.497934 9669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:54.502893 9674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:54.502910 9677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.502944 9669 server_base.cc:1034] running on GCE node
W20260928 06:34:54.502884 9675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.503363 9669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:54.503739 9669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:54.504936 9669 hybrid_clock.cc:648] HybridClock initialized: now 1790577294504924 us; error 35 us; skew 500 ppm
I20260928 06:34:54.506453 9669 webserver.cc:460] Webserver started at http://127.24.11.190:43011/ using document root <none> and password file <none>
I20260928 06:34:54.506726 9669 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:54.506803 9669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:54.506958 9669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:54.508051 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "09836eea3d9a459a888a2e6684f6c1e6"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "5cf62a4072c4d86d85197f95a9366970"
server_key_iv: "ded2a0ce24beff06bda7b48b4b578d4f"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.508503 9669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "09836eea3d9a459a888a2e6684f6c1e6"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "5cf62a4072c4d86d85197f95a9366970"
server_key_iv: "ded2a0ce24beff06bda7b48b4b578d4f"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.510452 9669 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:54.511354 9683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.511965 9669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:54.512070 9669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "09836eea3d9a459a888a2e6684f6c1e6"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "5cf62a4072c4d86d85197f95a9366970"
server_key_iv: "ded2a0ce24beff06bda7b48b4b578d4f"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.512166 9669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.529711 9669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.530054 9669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.530215 9669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.534456 9669 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:38205
I20260928 06:34:54.534515 9735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:38205 every 8 connection(s)
I20260928 06:34:54.534893 9669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:54.535902 9736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:54.538340 9736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: Bootstrap starting.
I20260928 06:34:54.538951 9736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:54.539255 9736 log.cc:824] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:54.539479 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9669
I20260928 06:34:54.539613 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:34:54.539803 24622 external_mini_cluster.cc:1428] Setting key 76dc006a58eef247af3355bf831c435a
I20260928 06:34:54.540136 9736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: No bootstrap required, opened a new log
I20260928 06:34:54.542092 9736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } }
I20260928 06:34:54.542228 9736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:54.542250 9736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09836eea3d9a459a888a2e6684f6c1e6, State: Initialized, Role: FOLLOWER
I20260928 06:34:54.542383 9736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } }
I20260928 06:34:54.542470 9736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:54.542493 9736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:54.542560 9736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:54.543118 9736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } }
I20260928 06:34:54.543201 9736 leader_election.cc:302] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09836eea3d9a459a888a2e6684f6c1e6; no voters:
I20260928 06:34:54.543413 9736 leader_election.cc:288] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:54.543454 9741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:54.543615 9741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [term 1 LEADER]: Becoming Leader. State: Replica: 09836eea3d9a459a888a2e6684f6c1e6, State: Running, Role: LEADER
I20260928 06:34:54.543687 9736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:54.543745 9741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } }
I20260928 06:34:54.544154 9741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09836eea3d9a459a888a2e6684f6c1e6. Latest consensus state: current_term: 1 leader_uuid: "09836eea3d9a459a888a2e6684f6c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } } }
I20260928 06:34:54.544252 9742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09836eea3d9a459a888a2e6684f6c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09836eea3d9a459a888a2e6684f6c1e6" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38205 } } }
I20260928 06:34:54.544351 9742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:54.544337 9741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:54.545063 9750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:54.545658 9750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:54.546741 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.540303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.24.11.190:38205)
0928 06:34:54.540525 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.540532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.540556 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:34:54.541328 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.541335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.541396 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.541537 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.541777 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.543267 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.544314 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.544940 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.545032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.546024 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.546027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.546027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:54.546206 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.546238 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:54.546248 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.546340 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.546444 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.546517 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260928 06:34:54.547355 9750 catalog_manager.cc:1347] Generated new cluster ID: 785e1e8adba84d14bd3fcbfe99839727
I20260928 06:34:54.547451 9750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:54.564198 9750 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:54.564778 9750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:54.569177 9750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: Generated new TSK 0
I20260928 06:34:54.569366 9750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:34:54.572058 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--builtin_ntp_servers=127.24.11.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:34:54.660854 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:54.661194 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:54.661237 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:54.662599 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:54.662671 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:54.662716 9760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:54.664109 9760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:54.664393 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:54.664705 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:54.669828 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:54.669811 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.670013 9760 server_base.cc:1034] running on GCE node
W20260928 06:34:54.669831 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.670372 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:54.670651 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:54.671839 9760 hybrid_clock.cc:648] HybridClock initialized: now 1790577294671827 us; error 30 us; skew 500 ppm
I20260928 06:34:54.673243 9760 webserver.cc:460] Webserver started at http://127.24.11.129:38841/ using document root <none> and password file <none>
I20260928 06:34:54.673519 9760 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:54.673601 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:54.673728 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:54.675143 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "384bdf67ddc442dcb866ea1f546643ba"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "8631c22aac45f36a663c6c9bf32aff37"
server_key_iv: "0d49c8e19ba240fbd6089c6d246e6424"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.675566 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "384bdf67ddc442dcb866ea1f546643ba"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "8631c22aac45f36a663c6c9bf32aff37"
server_key_iv: "0d49c8e19ba240fbd6089c6d246e6424"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.677218 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:54.678124 9774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.678324 9760 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:34:54.678414 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "384bdf67ddc442dcb866ea1f546643ba"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "8631c22aac45f36a663c6c9bf32aff37"
server_key_iv: "0d49c8e19ba240fbd6089c6d246e6424"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.678483 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.695338 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.695708 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.695858 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.696120 9760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:54.696518 9760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:54.696578 9760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.696604 9760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:54.696617 9760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.702742 9760 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:35509
I20260928 06:34:54.702792 9887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:35509 every 8 connection(s)
I20260928 06:34:54.703140 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:54.707736 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9760
I20260928 06:34:54.707825 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:54.708007 24622 external_mini_cluster.cc:1428] Setting key ac1be800866fd9404c1646b1d900d51d
I20260928 06:34:54.709033 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.703960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:50703 (local address 127.24.11.190:38205)
0928 06:34:54.704044 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.704047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.704212 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:34:54.704797 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.704798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.704812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.704864 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.705613 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.706097 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.707312 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.707496 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.707585 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.708680 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.708681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.708682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:54.708686 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.708703 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:54.708707 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.708755 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.708907 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.708936 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:34:54.709197 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--builtin_ntp_servers=127.24.11.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:34:54.709384 9890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.703870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38205 (local address 127.24.11.129:50703)
0928 06:34:54.704087 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.704125 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.704658 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.704929 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.704932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.704942 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:54.705504 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.705509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.706192 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.706193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.707207 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.707213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.707607 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.707608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.707767 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.708502 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:54.708503 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:54.708531 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:54.708555 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:54.708791 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:54.708798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:54.708802 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.708900 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.709012 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260928 06:34:54.710026 9888 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38205
I20260928 06:34:54.710146 9888 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:54.710350 9888 heartbeater.cc:511] Master 127.24.11.190:38205 requested a full tablet report, sending...
I20260928 06:34:54.711002 9700 ts_manager.cc:194] Registered new tserver with Master: 384bdf67ddc442dcb866ea1f546643ba (127.24.11.129:35509)
I20260928 06:34:54.711892 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.129:50703
W20260928 06:34:54.793514 9891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:54.793812 9891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:54.793849 9891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:54.795104 9891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:54.795179 9891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:54.795228 9891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:54.796680 9891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:54.796984 9891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:54.797315 9891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:54.802631 9899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:54.802631 9897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.802862 9891 server_base.cc:1034] running on GCE node
W20260928 06:34:54.802713 9896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.803150 9891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:54.803452 9891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:54.804606 9891 hybrid_clock.cc:648] HybridClock initialized: now 1790577294804586 us; error 38 us; skew 500 ppm
I20260928 06:34:54.806046 9891 webserver.cc:460] Webserver started at http://127.24.11.130:38891/ using document root <none> and password file <none>
I20260928 06:34:54.806308 9891 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:54.806358 9891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:54.806531 9891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:54.807677 9891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "f13ef41375734bdcbdf4447f40e4d136"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "fdd8b473d003e186b4f646a910aeec27"
server_key_iv: "a5974cb83ce252c9dc94a537a5226294"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.808039 9891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "f13ef41375734bdcbdf4447f40e4d136"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "fdd8b473d003e186b4f646a910aeec27"
server_key_iv: "a5974cb83ce252c9dc94a537a5226294"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.809401 9891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:54.810199 9905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.810405 9891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:54.810497 9891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "f13ef41375734bdcbdf4447f40e4d136"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "fdd8b473d003e186b4f646a910aeec27"
server_key_iv: "a5974cb83ce252c9dc94a537a5226294"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.810573 9891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.830181 9891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.830536 9891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.830706 9891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.830971 9891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:54.831436 9891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:54.831472 9891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.831496 9891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:54.831540 9891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.837576 9891 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33573
I20260928 06:34:54.837603 10018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33573 every 8 connection(s)
I20260928 06:34:54.837958 9891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:54.843241 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.838665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:51905 (local address 127.24.11.190:38205)
0928 06:34:54.838709 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.838712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.838971 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:34:54.839449 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.839451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.839461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.839517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.840167 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.840664 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.841773 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.841969 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.842020 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.842959 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.842961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.842961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:54.842988 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.843005 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:54.843010 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.843066 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.843154 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.843178 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:34:54.843448 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.838667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38205 (local address 127.24.11.130:51905)
0928 06:34:54.838839 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.838875 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.839359 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.839572 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.839575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.839584 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:54.840089 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.840092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.840780 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.840781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.841687 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.841690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.842048 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.842049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.842171 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.842822 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:54.842823 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:54.842853 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:54.842878 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:54.843081 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:54.843086 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:54.843088 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.843146 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.843239 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 06:34:54.843999 10019 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38205
I20260928 06:34:54.844125 10019 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:54.844321 10019 heartbeater.cc:511] Master 127.24.11.190:38205 requested a full tablet report, sending...
I20260928 06:34:54.844749 9700 ts_manager.cc:194] Registered new tserver with Master: f13ef41375734bdcbdf4447f40e4d136 (127.24.11.130:33573)
I20260928 06:34:54.844841 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 9891
I20260928 06:34:54.844956 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:54.845160 24622 external_mini_cluster.cc:1428] Setting key d7f29e59fa29cbac9edc6c833a84c60d
I20260928 06:34:54.845286 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.130:51905
I20260928 06:34:54.846192 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--builtin_ntp_servers=127.24.11.148:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:34:54.928324 10022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:54.928532 10022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:54.928555 10022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:54.929761 10022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:54.929804 10022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:54.929833 10022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:54.931279 10022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:54.931542 10022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:54.931843 10022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:54.936940 10027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.937004 10022 server_base.cc:1034] running on GCE node
W20260928 06:34:54.936949 10030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:54.936954 10028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:54.937393 10022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:54.937686 10022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:54.938872 10022 hybrid_clock.cc:648] HybridClock initialized: now 1790577294938854 us; error 27 us; skew 500 ppm
I20260928 06:34:54.940160 10022 webserver.cc:460] Webserver started at http://127.24.11.131:43015/ using document root <none> and password file <none>
I20260928 06:34:54.940418 10022 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:54.940490 10022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:54.940639 10022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:54.941819 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "de0e014077412a046685f9b3279ac43f"
server_key_iv: "00fcd6116b390de028f42b84c754a10d"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.942191 10022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "de0e014077412a046685f9b3279ac43f"
server_key_iv: "00fcd6116b390de028f42b84c754a10d"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.943651 10022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:54.944470 10036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.944769 10022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:54.944873 10022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707"
format_stamp: "Formatted at 2026-09-28 06:34:54 on dist-test-slave-38pg"
server_key: "de0e014077412a046685f9b3279ac43f"
server_key_iv: "00fcd6116b390de028f42b84c754a10d"
server_key_version: "encryptionkey@0"
I20260928 06:34:54.944963 10022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:54.962510 10022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:54.962870 10022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:54.963037 10022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:54.963290 10022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:54.963632 10022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:54.963697 10022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.963743 10022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:54.963768 10022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:54.969558 10022 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:34835
I20260928 06:34:54.969635 10149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:34835 every 8 connection(s)
I20260928 06:34:54.969995 10022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:54.971235 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10022
I20260928 06:34:54.971344 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:54.971529 24622 external_mini_cluster.cc:1428] Setting key f4242b6a5d6b002e4cafd3990db0ee15
I20260928 06:34:54.975581 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.970811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:40327 (local address 127.24.11.190:38205)
0928 06:34:54.970877 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.970879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.971169 (+ 290us) server_negotiation.cc:407] Connection header received
0928 06:34:54.971822 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.971823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.971835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.971883 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.972621 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.973124 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.974147 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.974350 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.974427 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.975279 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.975281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.975282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:54.975287 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.975313 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:54.975317 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.975377 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.975475 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.975503 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:34:54.975842 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.970807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38205 (local address 127.24.11.131:40327)
0928 06:34:54.971018 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.971092 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.971708 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.971966 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.971970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.971986 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:54.972531 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.972535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.973225 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.973226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.974055 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.974058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.974449 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.974450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.974585 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.975136 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:54.975137 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:54.975166 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:54.975190 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:54.975379 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:54.975385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:54.975388 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.975478 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.975583 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260928 06:34:54.976397 10150 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38205
I20260928 06:34:54.976476 10150 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:54.976662 10150 heartbeater.cc:511] Master 127.24.11.190:38205 requested a full tablet report, sending...
I20260928 06:34:54.977056 9700 ts_manager.cc:194] Registered new tserver with Master: 41070e3ccdbc40f1a8fc9bd9dd0fd707 (127.24.11.131:34835)
I20260928 06:34:54.977421 9700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.131:40327
I20260928 06:34:54.983407 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:54.987888 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.984817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.24.11.190:38205)
0928 06:34:54.984910 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.984912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.985013 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:34:54.985083 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.985083 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.985094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.985150 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:54.985337 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.985910 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.986667 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.986883 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.986961 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.987597 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:54.987599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:54.987599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:54.987604 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:54.987619 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:54.987623 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:54.987671 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.987734 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.987755 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:34:54.989444 9700 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:54.991081 9700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:54.998181 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33573 (local address 127.0.0.1:46960)
0928 06:34:54.993591 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.993600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.993663 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.993981 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.993982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.993983 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.994065 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.994067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.995435 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.995436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.996839 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.996840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.997853 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.997854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.997937 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.997952 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.998031 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.998088 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":72,"thread_start_us":82,"threads_started":1}
I20260928 06:34:54.998198 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.24.11.130:33573)
0928 06:34:54.993774 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.993780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.993793 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:34:54.993842 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.993843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.993857 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.993956 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.994138 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.995329 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.996943 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.997736 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.997806 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.997901 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.998062 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.998104 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260928 06:34:54.998733 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:35509 (local address 127.0.0.1:49980)
0928 06:34:54.993487 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.993735 (+ 248us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.993772 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.993973 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.993976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.993978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.994065 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.994068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.996490 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.996492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.997431 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.997434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.998494 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.998495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.998560 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.998564 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.998609 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.998649 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":8,"thread_start_us":73,"threads_started":1}
I20260928 06:34:54.999037 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.24.11.129:35509)
0928 06:34:54.993781 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.993785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.993794 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:54.993837 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.993839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.993855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.993949 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.995442 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.996400 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.997523 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.998277 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.998370 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.998461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.998913 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.998959 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":90,"threads_started":1}
I20260928 06:34:54.999269 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:34835 (local address 127.0.0.1:37862)
0928 06:34:54.993578 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:54.993593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:54.993663 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:54.994405 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:54.994406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:54.994407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:54.994442 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.994443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.995605 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.995606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.996836 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:54.996839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.998795 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.998796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:54.998884 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.998887 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:54.999141 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 06:34:54.999194 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260928 06:34:54.999315 9953 tablet_service.cc:1467] Processing CreateTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 (DEFAULT_TABLE table=test-table [id=d755ceb92df24bb29a7494b67f091cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:54.999539 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:54.993564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.24.11.131:34835)
0928 06:34:54.994256 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:54.994261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:54.994270 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:54.994307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:54.994309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:54.994325 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:54.994406 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:54.994495 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.995514 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.997589 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:54.998612 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:54.998674 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:54.998762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:54.999414 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 06:34:54.999457 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":88,"threads_started":1}
I20260928 06:34:54.999759 9953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496dd0063cd54f1ebe0ce0786f3db1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:55.000147 10084 tablet_service.cc:1467] Processing CreateTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 (DEFAULT_TABLE table=test-table [id=d755ceb92df24bb29a7494b67f091cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:55.000458 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496dd0063cd54f1ebe0ce0786f3db1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:55.002714 10164 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Bootstrap starting.
I20260928 06:34:55.003173 9822 tablet_service.cc:1467] Processing CreateTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 (DEFAULT_TABLE table=test-table [id=d755ceb92df24bb29a7494b67f091cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:55.003491 10164 tablet_bootstrap.cc:654] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:55.003500 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496dd0063cd54f1ebe0ce0786f3db1c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:55.003787 10164 log.cc:824] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:55.004495 10164 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: No bootstrap required, opened a new log
I20260928 06:34:55.004554 10164 ts_tablet_manager.cc:1397] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:55.005744 10165 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Bootstrap starting.
I20260928 06:34:55.006345 10167 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Bootstrap starting.
I20260928 06:34:55.006435 10165 tablet_bootstrap.cc:654] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:55.006765 10165 log.cc:824] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:55.007053 10167 tablet_bootstrap.cc:654] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:55.007370 10167 log.cc:824] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:55.007474 10164 raft_consensus.cc:348] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.007609 10165 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: No bootstrap required, opened a new log
I20260928 06:34:55.007650 10164 raft_consensus.cc:374] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:55.007674 10165 ts_tablet_manager.cc:1397] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:55.007687 10164 raft_consensus.cc:729] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13ef41375734bdcbdf4447f40e4d136, State: Initialized, Role: FOLLOWER
I20260928 06:34:55.007836 10164 consensus_queue.cc:260] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.008152 10167 tablet_bootstrap.cc:492] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: No bootstrap required, opened a new log
I20260928 06:34:55.008164 10019 heartbeater.cc:503] Master 127.24.11.190:38205 was elected leader, sending a full tablet report...
I20260928 06:34:55.008222 10167 ts_tablet_manager.cc:1397] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:55.008104 10164 ts_tablet_manager.cc:1428] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:34:55.010124 10165 raft_consensus.cc:348] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.010248 10165 raft_consensus.cc:374] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:55.010270 10165 raft_consensus.cc:729] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41070e3ccdbc40f1a8fc9bd9dd0fd707, State: Initialized, Role: FOLLOWER
I20260928 06:34:55.010394 10165 consensus_queue.cc:260] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.010660 10165 ts_tablet_manager.cc:1428] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:55.010705 10150 heartbeater.cc:503] Master 127.24.11.190:38205 was elected leader, sending a full tablet report...
I20260928 06:34:55.010628 10167 raft_consensus.cc:348] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.010788 10167 raft_consensus.cc:374] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:55.010824 10167 raft_consensus.cc:729] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384bdf67ddc442dcb866ea1f546643ba, State: Initialized, Role: FOLLOWER
I20260928 06:34:55.010931 10167 consensus_queue.cc:260] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.011186 10167 ts_tablet_manager.cc:1428] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:34:55.011278 9888 heartbeater.cc:503] Master 127.24.11.190:38205 was elected leader, sending a full tablet report...
W20260928 06:34:55.088526 10020 tablet.cc:2369] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:34:55.203889 9889 tablet.cc:2369] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:34:55.220572 10151 tablet.cc:2369] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:55.272346 10171 raft_consensus.cc:482] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:55.272521 10171 raft_consensus.cc:504] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.272905 10171 leader_election.cc:288] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 384bdf67ddc442dcb866ea1f546643ba (127.24.11.129:35509), f13ef41375734bdcbdf4447f40e4d136 (127.24.11.130:33573)
I20260928 06:34:55.276207 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.273297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42757 (local address 127.24.11.130:33573)
0928 06:34:55.273351 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:55.273353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:55.273508 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:34:55.273557 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:55.273557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:55.273567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:55.273625 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:55.273923 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.274394 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.275380 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.275807 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.275875 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:55.275885 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:55.276080 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:34:55.276128 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:34:55.276343 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.273301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:55219 (local address 127.24.11.129:35509)
0928 06:34:55.273348 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:55.273350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:55.273364 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:34:55.273395 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:55.273396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:55.273409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:55.273476 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:55.273716 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.274337 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.275445 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.275956 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.276031 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:55.276039 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:55.276221 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:34:55.276272 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:34:55.276363 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.273129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:35509 (local address 127.24.11.131:55219)
0928 06:34:55.273226 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:55.273249 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:55.273320 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:55.273498 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:55.273499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:55.273500 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:55.273605 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:55.273608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.274435 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.274436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:55.275332 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:55.275334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.276041 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.276042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:55.276142 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:55.276145 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:55.276202 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:34:55.276253 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":20}
I20260928 06:34:55.276487 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.273129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33573 (local address 127.24.11.131:42757)
0928 06:34:55.273392 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:55.273403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:55.273467 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:55.273711 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:55.273712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:55.273714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:55.273792 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:55.273794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.274480 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.274481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:55.275288 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:55.275290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.275898 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.275899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:55.275975 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:55.275979 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:55.276340 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 06:34:55.276389 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":97,"thread_start_us":93,"threads_started":1}
I20260928 06:34:55.276762 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dd0063cd54f1ebe0ce0786f3db1c7" candidate_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384bdf67ddc442dcb866ea1f546643ba" is_pre_election: true
I20260928 06:34:55.276958 9842 raft_consensus.cc:2454] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41070e3ccdbc40f1a8fc9bd9dd0fd707 in term 0.
I20260928 06:34:55.276932 9973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dd0063cd54f1ebe0ce0786f3db1c7" candidate_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13ef41375734bdcbdf4447f40e4d136" is_pre_election: true
I20260928 06:34:55.277113 9973 raft_consensus.cc:2454] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41070e3ccdbc40f1a8fc9bd9dd0fd707 in term 0.
I20260928 06:34:55.277238 10039 leader_election.cc:302] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384bdf67ddc442dcb866ea1f546643ba, 41070e3ccdbc40f1a8fc9bd9dd0fd707; no voters:
I20260928 06:34:55.277424 10171 raft_consensus.cc:2781] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:55.277468 10171 raft_consensus.cc:482] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:55.277485 10171 raft_consensus.cc:3037] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:55.278218 10171 raft_consensus.cc:504] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.278362 10171 leader_election.cc:288] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [CANDIDATE]: Term 1 election: Requested vote from peers 384bdf67ddc442dcb866ea1f546643ba (127.24.11.129:35509), f13ef41375734bdcbdf4447f40e4d136 (127.24.11.130:33573)
I20260928 06:34:55.278508 9842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dd0063cd54f1ebe0ce0786f3db1c7" candidate_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384bdf67ddc442dcb866ea1f546643ba"
I20260928 06:34:55.278596 9842 raft_consensus.cc:3037] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:55.279426 9842 raft_consensus.cc:2454] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41070e3ccdbc40f1a8fc9bd9dd0fd707 in term 1.
I20260928 06:34:55.279613 9973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dd0063cd54f1ebe0ce0786f3db1c7" candidate_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f13ef41375734bdcbdf4447f40e4d136"
I20260928 06:34:55.279678 10039 leader_election.cc:302] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384bdf67ddc442dcb866ea1f546643ba, 41070e3ccdbc40f1a8fc9bd9dd0fd707; no voters:
I20260928 06:34:55.279714 9973 raft_consensus.cc:3037] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:55.279774 10171 raft_consensus.cc:2781] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:55.279917 10171 raft_consensus.cc:686] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 1 LEADER]: Becoming Leader. State: Replica: 41070e3ccdbc40f1a8fc9bd9dd0fd707, State: Running, Role: LEADER
I20260928 06:34:55.280018 10171 consensus_queue.cc:237] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } }
I20260928 06:34:55.280694 9973 raft_consensus.cc:2454] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41070e3ccdbc40f1a8fc9bd9dd0fd707 in term 1.
I20260928 06:34:55.281199 9699 catalog_manager.cc:5495] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41070e3ccdbc40f1a8fc9bd9dd0fd707 (127.24.11.131). New cstate: current_term: 1 leader_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41070e3ccdbc40f1a8fc9bd9dd0fd707" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 } health_report { overall_health: UNKNOWN } } }
I20260928 06:34:55.350649 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.348451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.24.11.131:34835)
0928 06:34:55.348492 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:55.348494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:55.348502 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:55.348579 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:55.348580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:55.348598 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:55.348667 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:55.348827 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.349275 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.350004 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:55.350188 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:55.350238 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:55.350368 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:55.350465 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:55.350521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:55.350550 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:34:55.350565 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:34:55.352512 9842 raft_consensus.cc:1261] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 384bdf67ddc442dcb866ea1f546643ba [term 1 FOLLOWER]: Refusing update from remote peer 41070e3ccdbc40f1a8fc9bd9dd0fd707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:55.352524 9973 raft_consensus.cc:1261] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 1 FOLLOWER]: Refusing update from remote peer 41070e3ccdbc40f1a8fc9bd9dd0fd707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:55.352902 10171 consensus_queue.cc:1035] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384bdf67ddc442dcb866ea1f546643ba" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:55.353046 10174 consensus_queue.cc:1035] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13ef41375734bdcbdf4447f40e4d136" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:55.356719 10178 mvcc.cc:205] Tried to move back new op lower bound from 7334204601761132544 to 7334204601467559936. Current Snapshot: MvccSnapshot[applied={T|T < 7334204601761132544 or (T in {7334204601761132544})}]
I20260928 06:34:55.357906 9700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:55.358007 9700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:55.358779 9700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09836eea3d9a459a888a2e6684f6c1e6: Sending DeleteTablet for 3 replicas of tablet 496dd0063cd54f1ebe0ce0786f3db1c7
I20260928 06:34:55.359140 9822 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:55 UTC) from {username='slave'} at 127.0.0.1:49980
I20260928 06:34:55.359229 9953 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:55 UTC) from {username='slave'} at 127.0.0.1:46960
I20260928 06:34:55.359272 10084 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dd0063cd54f1ebe0ce0786f3db1c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:55 UTC) from {username='slave'} at 127.0.0.1:37862
I20260928 06:34:55.359562 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9760
W20260928 06:34:55.365268 9686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:35509 (error 108)
I20260928 06:34:55.365338 10188 tablet_replica.cc:330] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136: stopping tablet replica
I20260928 06:34:55.365460 10188 raft_consensus.cc:2229] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:55.365458 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9891
I20260928 06:34:55.365607 10188 raft_consensus.cc:2258] T 496dd0063cd54f1ebe0ce0786f3db1c7 P f13ef41375734bdcbdf4447f40e4d136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:55.365789 10189 tablet_replica.cc:330] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: stopping tablet replica
I20260928 06:34:55.365887 10189 raft_consensus.cc:2229] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:34:55.365989 10189 raft_consensus.cc:2258] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:55.366850 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.366721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:35509 (local address 127.0.0.1:49986)
0928 06:34:55.366772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:55.366808 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:35509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:34:55.367043 9686 catalog_manager.cc:4584] TS 384bdf67ddc442dcb866ea1f546643ba (127.24.11.129:35509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 496dd0063cd54f1ebe0ce0786f3db1c7: Network error: Client connection negotiation failed: client connection to 127.24.11.129:35509: connect: Connection refused (error 111)
I20260928 06:34:55.367246 10189 ts_tablet_manager.cc:1905] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:55.369189 10189 ts_tablet_manager.cc:1918] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:55.369268 10189 log.cc:1196] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/496dd0063cd54f1ebe0ce0786f3db1c7
I20260928 06:34:55.369519 10189 ts_tablet_manager.cc:1939] T 496dd0063cd54f1ebe0ce0786f3db1c7 P 41070e3ccdbc40f1a8fc9bd9dd0fd707: Deleting consensus metadata
I20260928 06:34:55.369985 9685 catalog_manager.cc:4841] TS 41070e3ccdbc40f1a8fc9bd9dd0fd707 (127.24.11.131:34835): tablet 496dd0063cd54f1ebe0ce0786f3db1c7 (table test-table [id=d755ceb92df24bb29a7494b67f091cbc]) successfully deleted
I20260928 06:34:55.371070 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10022
I20260928 06:34:55.371202 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:55.371073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33573 (local address 127.0.0.1:46968)
0928 06:34:55.371137 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:55.371166 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:33573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:34:55.376649 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 9669
2026-09-28T06:34:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:34:55 dist-test-slave-38pg krb5kdc[10193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:34:55 dist-test-slave-38pg krb5kdc[10193](info): set up 2 sockets
Sep 28 06:34:55 dist-test-slave-38pg krb5kdc[10193](info): commencing operation
krb5kdc: starting...
W20260928 06:34:56.422446 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:34:56.542244 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:46709
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:46709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:56.624083 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:56.624300 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:56.624323 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:56.625509 10206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:56.625574 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:56.625629 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:56.625658 10206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:56.625686 10206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:56.627269 10206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:46709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:46709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:56.627542 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:56.627805 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:56.632839 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.632839 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.632871 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:56.633003 10206 server_base.cc:1034] running on GCE node
I20260928 06:34:56.633435 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:56.633816 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:56.635001 10206 hybrid_clock.cc:648] HybridClock initialized: now 1790577296634987 us; error 49 us; skew 500 ppm
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:56.637696 10206 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:34:56.638096 10206 webserver.cc:460] Webserver started at http://127.24.11.190:34461/ using document root <none> and password file <none>
I20260928 06:34:56.638337 10206 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:56.638386 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:56.638520 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:56.639611 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "b2c0780f89d34b998e145db70bfe48fe"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "5a707702ce07130f0c7d8eea3e7a32e8"
server_key_iv: "56d25e2dab37c2bd79a9ac7c1562b069"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.639985 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "b2c0780f89d34b998e145db70bfe48fe"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "5a707702ce07130f0c7d8eea3e7a32e8"
server_key_iv: "56d25e2dab37c2bd79a9ac7c1562b069"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.641345 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:56.642223 10221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:56.642418 10206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:34:56.642520 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "b2c0780f89d34b998e145db70bfe48fe"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "5a707702ce07130f0c7d8eea3e7a32e8"
server_key_iv: "56d25e2dab37c2bd79a9ac7c1562b069"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.642604 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:56.653787 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:56.655707 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:56.655874 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:56.659893 10206 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:46709
I20260928 06:34:56.659929 10273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:46709 every 8 connection(s)
I20260928 06:34:56.660288 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:56.661275 10274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:56.663618 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: Bootstrap starting.
I20260928 06:34:56.664145 10274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:56.664371 10274 log.cc:824] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:56.665105 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: No bootstrap required, opened a new log
I20260928 06:34:56.666774 10274 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } }
I20260928 06:34:56.666909 10274 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:56.666930 10274 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c0780f89d34b998e145db70bfe48fe, State: Initialized, Role: FOLLOWER
I20260928 06:34:56.667081 10274 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } }
I20260928 06:34:56.667167 10274 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:56.667192 10274 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:56.667254 10274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:56.667282 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10206
I20260928 06:34:56.667451 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:34:56.667654 24622 external_mini_cluster.cc:1428] Setting key 705a5d28e42d39252657a4c0145018c2
I20260928 06:34:56.668036 10274 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } }
I20260928 06:34:56.668177 10274 leader_election.cc:302] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2c0780f89d34b998e145db70bfe48fe; no voters:
I20260928 06:34:56.668366 10274 leader_election.cc:288] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:56.668393 10278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:56.668535 10278 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [term 1 LEADER]: Becoming Leader. State: Replica: b2c0780f89d34b998e145db70bfe48fe, State: Running, Role: LEADER
I20260928 06:34:56.668635 10274 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:56.668664 10278 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } }
I20260928 06:34:56.668974 10278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c0780f89d34b998e145db70bfe48fe. Latest consensus state: current_term: 1 leader_uuid: "b2c0780f89d34b998e145db70bfe48fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } } }
I20260928 06:34:56.669234 10278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:56.669553 10288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:56.669788 10280 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2c0780f89d34b998e145db70bfe48fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2c0780f89d34b998e145db70bfe48fe" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 46709 } } }
I20260928 06:34:56.669960 10280 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:56.670217 10288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:56.671911 10288 catalog_manager.cc:1347] Generated new cluster ID: a150f820877244b9b2fad9c91c85adae
I20260928 06:34:56.671963 10288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:56.678807 10288 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:56.679487 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:56.668209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.24.11.190:46709)
0928 06:34:56.668555 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:56.668564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:56.668596 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:34:56.668722 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:56.668730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:56.668835 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:56.668985 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:56.669956 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.671400 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.672427 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.672901 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.673031 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:56.675592 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:56.675597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:56.675597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:56.675609 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:56.677577 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:56.677976 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:56.677979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:56.677980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:56.678092 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:56.678343 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:56.678344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:56.678345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:56.678636 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:56.678847 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:56.679135 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:34:56.679301 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":120,"threads_started":1}
I20260928 06:34:56.679621 10288 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:56.683734 10288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: Generated new TSK 0
I20260928 06:34:56.683897 10288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:34:56.713068 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:56.797135 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:56.797390 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:56.797435 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:56.798738 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:56.798825 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:56.798871 10300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:56.800408 10300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:56.800699 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:56.800961 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:56.806131 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.806157 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.806188 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:56.806563 10300 server_base.cc:1034] running on GCE node
I20260928 06:34:56.806777 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:56.806993 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:56.808138 10300 hybrid_clock.cc:648] HybridClock initialized: now 1790577296808125 us; error 26 us; skew 500 ppm
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:56.810914 10300 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:34:56.811328 10300 webserver.cc:460] Webserver started at http://127.24.11.129:44955/ using document root <none> and password file <none>
I20260928 06:34:56.811532 10300 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:56.811573 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:56.811663 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:56.812707 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "8d6cab94963249ae8da00adee718d1a5"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "4d7fa343ac5e782ec7fd97588c47da67"
server_key_iv: "52217e13ffa54813634e9f0aed190897"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.813009 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "8d6cab94963249ae8da00adee718d1a5"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "4d7fa343ac5e782ec7fd97588c47da67"
server_key_iv: "52217e13ffa54813634e9f0aed190897"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.814358 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:56.815052 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:56.815248 10300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:56.815339 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "8d6cab94963249ae8da00adee718d1a5"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "4d7fa343ac5e782ec7fd97588c47da67"
server_key_iv: "52217e13ffa54813634e9f0aed190897"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.815397 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:56.844475 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:56.847136 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:56.847257 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:56.847455 10300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:56.847782 10300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:56.847826 10300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:56.847874 10300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:56.847899 10300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:56.853819 10300 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:39281
I20260928 06:34:56.853857 10428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:39281 every 8 connection(s)
I20260928 06:34:56.854216 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:56.858755 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10300
I20260928 06:34:56.858899 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:56.859133 24622 external_mini_cluster.cc:1428] Setting key 6755896986745204edd7bd72a66df04d
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:56.864161 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:56.855066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:47857 (local address 127.24.11.190:46709)
0928 06:34:56.855161 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:56.855164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:56.855289 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:34:56.855752 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:56.855753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:56.855763 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:56.855819 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:56.856744 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.857237 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.858405 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.858621 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.858677 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:56.861226 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:56.861227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:56.861228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:56.861237 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:56.862728 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:56.863193 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:56.863194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:56.863194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:56.863254 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:56.863511 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:56.863512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:56.863512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:56.863628 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:56.863704 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:56.863982 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 06:34:56.864067 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:34:56.864375 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:56.854964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46709 (local address 127.24.11.129:47857)
0928 06:34:56.855150 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:56.855190 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:56.855667 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:56.855879 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:56.855882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:56.856463 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:56.856662 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:56.856667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.857340 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.857341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:56.858315 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:56.858318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:56.858714 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:56.858715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:56.858873 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:56.859515 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:56.859533 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:56.861116 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:56.862853 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:56.862860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:56.862862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:56.863099 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:56.863326 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:56.863327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:56.863328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:56.863444 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:56.863696 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:56.863703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:56.863850 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 06:34:56.863962 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 06:34:56.864129 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260928 06:34:56.864895 10429 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46709
I20260928 06:34:56.865011 10429 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:56.865195 10429 heartbeater.cc:511] Master 127.24.11.190:46709 requested a full tablet report, sending...
I20260928 06:34:56.865770 10238 ts_manager.cc:194] Registered new tserver with Master: 8d6cab94963249ae8da00adee718d1a5 (127.24.11.129:39281)
I20260928 06:34:56.866379 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:47857
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:34:56.884622 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:56.967024 10434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:56.967259 10434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:56.967286 10434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:56.968525 10434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:56.968570 10434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:56.968600 10434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:56.970219 10434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:56.970467 10434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:56.970708 10434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:56.976084 10439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.976080 10442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:56.976055 10440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:56.976125 10434 server_base.cc:1034] running on GCE node
I20260928 06:34:56.976573 10434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:56.976840 10434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:56.978008 10434 hybrid_clock.cc:648] HybridClock initialized: now 1790577296978004 us; error 42 us; skew 500 ppm
Sep 28 06:34:56 dist-test-slave-38pg krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:56.980698 10434 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:34:56.981154 10434 webserver.cc:460] Webserver started at http://127.24.11.130:42877/ using document root <none> and password file <none>
I20260928 06:34:56.981383 10434 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:56.981428 10434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:56.981670 10434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:56.982760 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "1dc6614d81544a799da5016769e6ca3e"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "0040034426b1cf2220a54ceeb26d96d0"
server_key_iv: "77f4fee2bf1e2893b1606c98de879b96"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.983134 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "1dc6614d81544a799da5016769e6ca3e"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "0040034426b1cf2220a54ceeb26d96d0"
server_key_iv: "77f4fee2bf1e2893b1606c98de879b96"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.984506 10434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:56.985351 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:56.985570 10434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:56.985662 10434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "1dc6614d81544a799da5016769e6ca3e"
format_stamp: "Formatted at 2026-09-28 06:34:56 on dist-test-slave-38pg"
server_key: "0040034426b1cf2220a54ceeb26d96d0"
server_key_iv: "77f4fee2bf1e2893b1606c98de879b96"
server_key_version: "encryptionkey@0"
I20260928 06:34:56.985719 10434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:56.998966 10434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.001422 10434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.001724 10434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.001962 10434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:57.002323 10434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:57.002379 10434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.002404 10434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:57.002419 10434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.009397 10434 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33431
I20260928 06:34:57.009430 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33431 every 8 connection(s)
I20260928 06:34:57.009855 10434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:34:57 dist-test-slave-38pg krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577296, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:57.019640 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10434
I20260928 06:34:57.019779 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:57.019781 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.010733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:46065 (local address 127.24.11.190:46709)
0928 06:34:57.010874 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.010877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.011038 (+ 161us) server_negotiation.cc:407] Connection header received
0928 06:34:57.011588 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.011589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.011602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.011648 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.012615 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.013130 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.014195 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.014391 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.014479 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.016702 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.016704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.016704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:57.016712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.018431 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.018798 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.018798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.018799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.018852 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.019073 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.019073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.019073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.019187 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.019284 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.019550 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.019659 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 06:34:57.019863 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.010648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46709 (local address 127.24.11.130:46065)
0928 06:34:57.010900 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.010946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.011473 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.011734 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.011738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.012321 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:57.012514 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.012519 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.013228 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.013229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.014093 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.014096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.014504 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.014505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.014626 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.015158 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:57.015166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:57.016605 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:57.018506 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:57.018512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:57.018514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:57.018729 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:57.018913 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:57.018913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:57.018914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:57.019014 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:57.019273 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:57.019279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:57.019413 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.019501 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.019635 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260928 06:34:57.020046 24622 external_mini_cluster.cc:1428] Setting key 2a6a296e0c9be5080a8f66c49847bcfa
I20260928 06:34:57.020452 10563 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46709
I20260928 06:34:57.020576 10563 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:57.020771 10563 heartbeater.cc:511] Master 127.24.11.190:46709 requested a full tablet report, sending...
I20260928 06:34:57.021169 10238 ts_manager.cc:194] Registered new tserver with Master: 1dc6614d81544a799da5016769e6ca3e (127.24.11.130:33431)
I20260928 06:34:57.021601 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:46065
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:34:57.044471 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:57.130221 10568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:57.130434 10568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:57.130458 10568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:57.131620 10568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:57.131668 10568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:57.131695 10568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:57.133155 10568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:46709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:57.133409 10568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:57.133672 10568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:57.138994 10576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.139056 10568 server_base.cc:1034] running on GCE node
W20260928 06:34:57.138998 10574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.138998 10573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.139489 10568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:57.139803 10568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:57.140995 10568 hybrid_clock.cc:648] HybridClock initialized: now 1790577297140981 us; error 30 us; skew 500 ppm
Sep 28 06:34:57 dist-test-slave-38pg krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577297, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:57.143646 10568 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:34:57.144042 10568 webserver.cc:460] Webserver started at http://127.24.11.131:33549/ using document root <none> and password file <none>
I20260928 06:34:57.144237 10568 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:57.144279 10568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:57.144369 10568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:57.145457 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "9a5fea0b294f481591da55f3a1574fbe"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "ecb30493dace99a97d23ff49a12c3171"
server_key_iv: "b0e6757b36c2e05e888b52af78891adf"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.145821 10568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "9a5fea0b294f481591da55f3a1574fbe"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "ecb30493dace99a97d23ff49a12c3171"
server_key_iv: "b0e6757b36c2e05e888b52af78891adf"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.147104 10568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:57.147791 10583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.148067 10568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:57.148134 10568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "9a5fea0b294f481591da55f3a1574fbe"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "ecb30493dace99a97d23ff49a12c3171"
server_key_iv: "b0e6757b36c2e05e888b52af78891adf"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.148180 10568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:57.161057 10568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.163348 10568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.163462 10568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.163645 10568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:57.163962 10568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:57.163997 10568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.164021 10568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:57.164064 10568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.170161 10568 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:42275
I20260928 06:34:57.170239 10696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:42275 every 8 connection(s)
I20260928 06:34:57.170717 10568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:34:57 dist-test-slave-38pg krb5kdc[10193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577297, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:57.179522 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10568
I20260928 06:34:57.179644 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:57.179847 24622 external_mini_cluster.cc:1428] Setting key c6992eb9f0e4b3835709d5638b061b5b
I20260928 06:34:57.180402 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.171520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:49597 (local address 127.24.11.190:46709)
0928 06:34:57.171576 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.171578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.171807 (+ 229us) server_negotiation.cc:407] Connection header received
0928 06:34:57.172373 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.172374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.172387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.172448 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.173371 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.173941 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.174941 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.175149 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.175223 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.177336 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.177338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.177338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:57.177346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.178624 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.179081 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.179082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.179082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.179145 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.179442 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.179443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.179443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.179625 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.179717 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.180180 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.180301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:34:57.180603 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.171512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:46709 (local address 127.24.11.131:49597)
0928 06:34:57.171689 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.171725 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.172262 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.172521 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.172524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.173087 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:57.173273 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.173278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.174033 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.174034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.174842 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.174844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.175229 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.175230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.175358 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.175901 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:57.175909 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:57.177226 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:57.178722 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:57.178729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:57.178731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:57.178992 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:57.179212 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:57.179213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:57.179213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:57.179322 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:57.179740 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:57.179747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:57.179926 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.180123 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.180347 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260928 06:34:57.181135 10697 heartbeater.cc:347] Connected to a master server at 127.24.11.190:46709
I20260928 06:34:57.181245 10697 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:57.181427 10697 heartbeater.cc:511] Master 127.24.11.190:46709 requested a full tablet report, sending...
I20260928 06:34:57.181825 10238 ts_manager.cc:194] Registered new tserver with Master: 9a5fea0b294f481591da55f3a1574fbe (127.24.11.131:42275)
I20260928 06:34:57.182163 10238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:49597
I20260928 06:34:57.191826 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:57.199491 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.193175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.24.11.190:46709)
0928 06:34:57.193264 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.193266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.193336 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:34:57.193377 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.193377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.193386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.193435 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.194026 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.194507 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.195189 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.195380 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.195432 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.196838 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.196839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.196839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:57.196846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.198259 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.198660 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.198661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.198662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.198715 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:57.198951 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:57.198952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:57.198952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:57.199085 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.199167 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.199300 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.199356 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:34:57.201227 10238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:57.202915 10238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:57.209329 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.24.11.130:33431)
0928 06:34:57.205784 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.205787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.205795 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:57.205828 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.205830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.205842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.205936 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.206114 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.206908 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.208277 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.208906 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.208992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209053 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.209206 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.209247 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260928 06:34:57.209318 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33431 (local address 127.0.0.1:41216)
0928 06:34:57.205514 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.205524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.205605 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.205943 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.205946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.205949 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.206030 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.206032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.207010 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.207011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.208180 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.208183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.208991 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.208992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.209070 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209083 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.209187 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.209242 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":77,"threads_started":1}
I20260928 06:34:57.209733 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:42275 (local address 127.0.0.1:43506)
0928 06:34:57.205714 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.205724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.205763 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.206079 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.206080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.206084 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.206141 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.206143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.207467 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.207468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.208562 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.208565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.209472 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.209473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.209553 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209557 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.209610 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.209649 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":31,"thread_start_us":195,"threads_started":1}
I20260928 06:34:57.209892 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.24.11.131:42275)
0928 06:34:57.205881 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.205888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.205896 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:34:57.205932 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.205934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.205948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.206015 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.206224 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.207355 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.208643 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.209336 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.209396 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209443 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.209754 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.209798 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":111,"threads_started":1}
I20260928 06:34:57.209932 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39281 (local address 127.0.0.1:45904)
0928 06:34:57.205464 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.205506 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.205600 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.206078 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.206079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.206081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.206141 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.206144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.207494 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.207495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.208371 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.208374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.209352 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.209354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.209641 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209645 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.209692 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.209724 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":5,"thread_start_us":151,"threads_started":1}
I20260928 06:34:57.209925 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.205336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.24.11.129:39281)
0928 06:34:57.205772 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.205778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.205789 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:34:57.205845 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.205847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.205863 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.205938 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.206231 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.207355 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.208466 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.209192 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.209263 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.209314 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.209777 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.209842 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":77,"threads_started":1}
I20260928 06:34:57.210337 10497 tablet_service.cc:1467] Processing CreateTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 (DEFAULT_TABLE table=test-table [id=08ecd596a529465d801dce6de114b4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.210559 10631 tablet_service.cc:1467] Processing CreateTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 (DEFAULT_TABLE table=test-table [id=08ecd596a529465d801dce6de114b4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.210709 10497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.210623 10363 tablet_service.cc:1467] Processing CreateTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 (DEFAULT_TABLE table=test-table [id=08ecd596a529465d801dce6de114b4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.210857 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.210949 10363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.213660 10714 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Bootstrap starting.
I20260928 06:34:57.214051 10716 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: Bootstrap starting.
I20260928 06:34:57.214680 10716 tablet_bootstrap.cc:654] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.214680 10714 tablet_bootstrap.cc:654] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.215338 10716 log.cc:824] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.215370 10714 log.cc:824] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.215687 10715 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Bootstrap starting.
I20260928 06:34:57.216145 10716 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: No bootstrap required, opened a new log
I20260928 06:34:57.216233 10716 ts_tablet_manager.cc:1397] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:57.216301 10715 tablet_bootstrap.cc:654] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.216379 10714 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: No bootstrap required, opened a new log
I20260928 06:34:57.216452 10714 ts_tablet_manager.cc:1397] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:34:57.216881 10715 log.cc:824] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.217737 10715 tablet_bootstrap.cc:492] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: No bootstrap required, opened a new log
I20260928 06:34:57.217818 10715 ts_tablet_manager.cc:1397] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:34:57.218307 10716 raft_consensus.cc:348] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.218467 10716 raft_consensus.cc:374] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.218509 10716 raft_consensus.cc:729] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6cab94963249ae8da00adee718d1a5, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.218643 10716 consensus_queue.cc:260] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.218691 10714 raft_consensus.cc:348] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.218864 10714 raft_consensus.cc:374] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.218927 10714 raft_consensus.cc:729] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5fea0b294f481591da55f3a1574fbe, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.218953 10716 ts_tablet_manager.cc:1428] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:57.219072 10429 heartbeater.cc:503] Master 127.24.11.190:46709 was elected leader, sending a full tablet report...
I20260928 06:34:57.219063 10714 consensus_queue.cc:260] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.219331 10714 ts_tablet_manager.cc:1428] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:57.219445 10697 heartbeater.cc:503] Master 127.24.11.190:46709 was elected leader, sending a full tablet report...
I20260928 06:34:57.219914 10715 raft_consensus.cc:348] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.220088 10715 raft_consensus.cc:374] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.220139 10715 raft_consensus.cc:729] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc6614d81544a799da5016769e6ca3e, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.220256 10715 consensus_queue.cc:260] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.220522 10715 ts_tablet_manager.cc:1428] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:57.220578 10563 heartbeater.cc:503] Master 127.24.11.190:46709 was elected leader, sending a full tablet report...
W20260928 06:34:57.260501 10564 tablet.cc:2369] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:57.279034 10721 raft_consensus.cc:482] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:57.279210 10721 raft_consensus.cc:504] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.279593 10721 leader_election.cc:288] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dc6614d81544a799da5016769e6ca3e (127.24.11.130:33431), 8d6cab94963249ae8da00adee718d1a5 (127.24.11.129:39281)
I20260928 06:34:57.282517 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.279891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:35425 (local address 127.24.11.129:39281)
0928 06:34:57.279944 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.279947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.279956 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:57.279987 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.279988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.279997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.280064 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.280252 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.280731 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.281752 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.282133 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.282227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.282236 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.282411 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.282453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:34:57.282517 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.279812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39281 (local address 127.24.11.131:35425)
0928 06:34:57.279872 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.279883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.279925 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.280069 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.280070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.280072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.280140 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.280142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.280819 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.280820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.281636 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.281643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.282227 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.282228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.282309 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.282312 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.282398 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.282442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:34:57.282624 10712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.279948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:45197 (local address 127.24.11.130:33431)
0928 06:34:57.279992 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.279994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.280101 (+ 107us) server_negotiation.cc:407] Connection header received
0928 06:34:57.280158 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.280159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.280169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.280248 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.280403 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.280889 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.281887 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.282260 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.282344 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.282352 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.282511 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.282550 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:34:57.282666 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.279853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33431 (local address 127.24.11.131:45197)
0928 06:34:57.280030 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.280038 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.280102 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.280251 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.280252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.280254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.280330 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.280332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.280974 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.280975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.281801 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.281803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.282357 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.282358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.282432 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.282435 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.282493 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.282532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":120,"threads_started":1}
I20260928 06:34:57.282913 10383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a9f4b1e35e471dbbdf89cbe4fcde01" candidate_uuid: "9a5fea0b294f481591da55f3a1574fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6cab94963249ae8da00adee718d1a5" is_pre_election: true
I20260928 06:34:57.283066 10383 raft_consensus.cc:2454] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5fea0b294f481591da55f3a1574fbe in term 0.
I20260928 06:34:57.283046 10517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a9f4b1e35e471dbbdf89cbe4fcde01" candidate_uuid: "9a5fea0b294f481591da55f3a1574fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc6614d81544a799da5016769e6ca3e" is_pre_election: true
I20260928 06:34:57.283249 10517 raft_consensus.cc:2454] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a5fea0b294f481591da55f3a1574fbe in term 0.
I20260928 06:34:57.283341 10586 leader_election.cc:302] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6cab94963249ae8da00adee718d1a5, 9a5fea0b294f481591da55f3a1574fbe; no voters:
I20260928 06:34:57.283497 10721 raft_consensus.cc:2781] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:57.283564 10721 raft_consensus.cc:482] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:57.283592 10721 raft_consensus.cc:3037] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:57.284307 10721 raft_consensus.cc:504] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.284444 10721 leader_election.cc:288] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [CANDIDATE]: Term 1 election: Requested vote from peers 1dc6614d81544a799da5016769e6ca3e (127.24.11.130:33431), 8d6cab94963249ae8da00adee718d1a5 (127.24.11.129:39281)
I20260928 06:34:57.284610 10383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a9f4b1e35e471dbbdf89cbe4fcde01" candidate_uuid: "9a5fea0b294f481591da55f3a1574fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6cab94963249ae8da00adee718d1a5"
I20260928 06:34:57.284615 10517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a9f4b1e35e471dbbdf89cbe4fcde01" candidate_uuid: "9a5fea0b294f481591da55f3a1574fbe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc6614d81544a799da5016769e6ca3e"
I20260928 06:34:57.284685 10383 raft_consensus.cc:3037] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:57.284749 10517 raft_consensus.cc:3037] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:57.285465 10383 raft_consensus.cc:2454] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5fea0b294f481591da55f3a1574fbe in term 1.
I20260928 06:34:57.285477 10517 raft_consensus.cc:2454] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a5fea0b294f481591da55f3a1574fbe in term 1.
I20260928 06:34:57.285648 10586 leader_election.cc:302] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d6cab94963249ae8da00adee718d1a5, 9a5fea0b294f481591da55f3a1574fbe; no voters:
I20260928 06:34:57.285755 10721 raft_consensus.cc:2781] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:57.285898 10721 raft_consensus.cc:686] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 1 LEADER]: Becoming Leader. State: Replica: 9a5fea0b294f481591da55f3a1574fbe, State: Running, Role: LEADER
I20260928 06:34:57.286103 10721 consensus_queue.cc:237] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } }
I20260928 06:34:57.287051 10237 catalog_manager.cc:5495] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a5fea0b294f481591da55f3a1574fbe (127.24.11.131). New cstate: current_term: 1 leader_uuid: "9a5fea0b294f481591da55f3a1574fbe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a5fea0b294f481591da55f3a1574fbe" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 42275 } health_report { overall_health: HEALTHY } } }
I20260928 06:34:57.301796 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.299384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.24.11.131:42275)
0928 06:34:57.299444 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.299446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.299456 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:57.299528 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.299528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.299549 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.299635 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:57.299758 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.300232 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.300977 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.301187 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.301261 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.301398 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:57.301495 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:57.301648 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.301698 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.301711 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:34:57.303773 10383 raft_consensus.cc:1261] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 8d6cab94963249ae8da00adee718d1a5 [term 1 FOLLOWER]: Refusing update from remote peer 9a5fea0b294f481591da55f3a1574fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:57.303792 10517 raft_consensus.cc:1261] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 1 FOLLOWER]: Refusing update from remote peer 9a5fea0b294f481591da55f3a1574fbe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:57.304164 10724 consensus_queue.cc:1035] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6cab94963249ae8da00adee718d1a5" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:57.304276 10721 consensus_queue.cc:1035] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc6614d81544a799da5016769e6ca3e" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:57.306783 10727 mvcc.cc:205] Tried to move back new op lower bound from 7334204609753137152 to 7334204609684848640. Current Snapshot: MvccSnapshot[applied={T|T < 7334204609753137152}]
I20260928 06:34:57.307202 10729 mvcc.cc:205] Tried to move back new op lower bound from 7334204609753137152 to 7334204609684848640. Current Snapshot: MvccSnapshot[applied={T|T < 7334204609753137152 or (T in {7334204609753137152})}]
I20260928 06:34:57.307618 10728 mvcc.cc:205] Tried to move back new op lower bound from 7334204609753137152 to 7334204609684848640. Current Snapshot: MvccSnapshot[applied={T|T < 7334204609753137152}]
I20260928 06:34:57.309861 10237 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:57.309944 10237 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:57.310720 10237 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2c0780f89d34b998e145db70bfe48fe: Sending DeleteTablet for 3 replicas of tablet 32a9f4b1e35e471dbbdf89cbe4fcde01
I20260928 06:34:57.311028 10497 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:57 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:41216
I20260928 06:34:57.311057 10363 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:57 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:45904
I20260928 06:34:57.311144 10631 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:57 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:43506
I20260928 06:34:57.311299 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10300
I20260928 06:34:57.313458 10739 tablet_replica.cc:330] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: stopping tablet replica
I20260928 06:34:57.313584 10739 raft_consensus.cc:2229] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:34:57.313719 10739 raft_consensus.cc:2258] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:57.314180 10739 ts_tablet_manager.cc:1905] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:57.315841 10739 ts_tablet_manager.cc:1918] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:57.315901 10739 log.cc:1196] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/32a9f4b1e35e471dbbdf89cbe4fcde01
I20260928 06:34:57.316186 10739 ts_tablet_manager.cc:1939] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 9a5fea0b294f481591da55f3a1574fbe: Deleting consensus metadata
I20260928 06:34:57.316561 10222 catalog_manager.cc:4841] TS 9a5fea0b294f481591da55f3a1574fbe (127.24.11.131:42275): tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 (table test-table [id=08ecd596a529465d801dce6de114b4ce]) successfully deleted
I20260928 06:34:57.317672 10737 tablet_replica.cc:330] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: stopping tablet replica
I20260928 06:34:57.317791 10737 raft_consensus.cc:2229] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:57.317904 10737 raft_consensus.cc:2258] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:57.318395 10737 ts_tablet_manager.cc:1905] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:57.320487 10737 ts_tablet_manager.cc:1918] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:57.320571 10737 log.cc:1196] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/32a9f4b1e35e471dbbdf89cbe4fcde01
I20260928 06:34:57.320860 10737 ts_tablet_manager.cc:1939] T 32a9f4b1e35e471dbbdf89cbe4fcde01 P 1dc6614d81544a799da5016769e6ca3e: Deleting consensus metadata
I20260928 06:34:57.321344 10225 catalog_manager.cc:4841] TS 1dc6614d81544a799da5016769e6ca3e (127.24.11.130:33431): tablet 32a9f4b1e35e471dbbdf89cbe4fcde01 (table test-table [id=08ecd596a529465d801dce6de114b4ce]) successfully deleted
W20260928 06:34:57.322264 10224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:39281 (error 108)
I20260928 06:34:57.322463 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10434
I20260928 06:34:57.324851 10709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.324681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:39281 (local address 127.0.0.1:45906)
0928 06:34:57.324759 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.324810 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 06:34:57.324991 10224 catalog_manager.cc:4584] TS 8d6cab94963249ae8da00adee718d1a5 (127.24.11.129:39281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32a9f4b1e35e471dbbdf89cbe4fcde01: Network error: Client connection negotiation failed: client connection to 127.24.11.129:39281: connect: Connection refused (error 111)
I20260928 06:34:57.328302 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10568
I20260928 06:34:57.332479 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10206
2026-09-28T06:34:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1956 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:57Z Disabled control of system clock
I20260928 06:34:57.359818 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38171
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:34:57.441007 10744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:57.441229 10744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:57.441250 10744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:57.442425 10744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:57.442468 10744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:57.442481 10744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:57.442494 10744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:57.442543 10744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:57.443933 10744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:38171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:57.444180 10744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:57.444480 10744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:57.449199 10750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.449205 10752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.449234 10749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.449465 10744 server_base.cc:1034] running on GCE node
I20260928 06:34:57.449709 10744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:57.450021 10744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:57.451174 10744 hybrid_clock.cc:648] HybridClock initialized: now 1790577297451162 us; error 43 us; skew 500 ppm
I20260928 06:34:57.452423 10744 webserver.cc:460] Webserver started at http://127.24.11.190:41941/ using document root <none> and password file <none>
I20260928 06:34:57.452730 10744 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:57.452790 10744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:57.452981 10744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:57.454186 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "abf82d2d739048b6b75291155db33a12"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "509ec2bbc6343c4433568ff99e143ea0"
server_key_iv: "904019f9e7fd06c6612712ddec22fe9d"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.454550 10744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "abf82d2d739048b6b75291155db33a12"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "509ec2bbc6343c4433568ff99e143ea0"
server_key_iv: "904019f9e7fd06c6612712ddec22fe9d"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.455853 10744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:34:57.456586 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.456796 10744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:34:57.456910 10744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "abf82d2d739048b6b75291155db33a12"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "509ec2bbc6343c4433568ff99e143ea0"
server_key_iv: "904019f9e7fd06c6612712ddec22fe9d"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.456986 10744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:57.465972 10744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.466290 10744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.466442 10744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.470633 10744 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:38171
I20260928 06:34:57.470660 10810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:38171 every 8 connection(s)
I20260928 06:34:57.471016 10744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:57.471969 10811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.474268 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10744
I20260928 06:34:57.474365 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:34:57.474352 10811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: Bootstrap starting.
I20260928 06:34:57.474546 24622 external_mini_cluster.cc:1428] Setting key 7ab4e891ec1e166e197ca5d3b43e148a
I20260928 06:34:57.475097 10811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.475433 10811 log.cc:824] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.476166 10811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: No bootstrap required, opened a new log
I20260928 06:34:57.478564 10811 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } }
I20260928 06:34:57.478725 10811 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.478754 10811 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf82d2d739048b6b75291155db33a12, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.478880 10811 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } }
I20260928 06:34:57.478952 10811 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:57.478981 10811 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:57.479027 10811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:57.479660 10811 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } }
I20260928 06:34:57.479779 10811 leader_election.cc:302] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf82d2d739048b6b75291155db33a12; no voters:
I20260928 06:34:57.479978 10811 leader_election.cc:288] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:57.480038 10816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:57.480188 10816 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [term 1 LEADER]: Becoming Leader. State: Replica: abf82d2d739048b6b75291155db33a12, State: Running, Role: LEADER
I20260928 06:34:57.480265 10811 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:57.480371 10816 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } }
I20260928 06:34:57.480716 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf82d2d739048b6b75291155db33a12. Latest consensus state: current_term: 1 leader_uuid: "abf82d2d739048b6b75291155db33a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } } }
I20260928 06:34:57.480810 10818 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf82d2d739048b6b75291155db33a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf82d2d739048b6b75291155db33a12" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 38171 } } }
I20260928 06:34:57.480875 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:57.480882 10818 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:57.480901 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.475124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.24.11.190:38171)
0928 06:34:57.475351 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.475362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.475396 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:34:57.475920 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.475929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.476006 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.476156 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.476377 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.477794 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.478556 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.479146 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.479268 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.479904 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.479908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.479909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:57.480128 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.480167 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:57.480184 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.480527 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.480558 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.480634 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260928 06:34:57.481159 10825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:57.481819 10825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:57.483501 10825 catalog_manager.cc:1347] Generated new cluster ID: b1524ba487ba4f0bb255d926c9f8cf23
I20260928 06:34:57.483557 10825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:34:57.490540 10825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:57.491055 10825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:57.495217 10825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: Generated new TSK 0
I20260928 06:34:57.495388 10825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:34:57.499054 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--builtin_ntp_servers=127.24.11.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:34:57.582741 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:57.582978 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:57.583014 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:57.584218 10835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:57.584277 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:57.584316 10835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:57.585696 10835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:57.585943 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:57.586226 10835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:57.591121 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.591135 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.591130 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.591544 10835 server_base.cc:1034] running on GCE node
I20260928 06:34:57.591758 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:57.592003 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:57.593165 10835 hybrid_clock.cc:648] HybridClock initialized: now 1790577297593152 us; error 30 us; skew 500 ppm
I20260928 06:34:57.594447 10835 webserver.cc:460] Webserver started at http://127.24.11.129:35959/ using document root <none> and password file <none>
I20260928 06:34:57.594694 10835 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:57.594763 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:57.594914 10835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:57.595954 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "0a414f19f41449c8be5976f2b60d33ad"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "35b96545dccc9e4462b1e68020761aba"
server_key_iv: "bf0725c1b52def164e8a0f908145a8c2"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.596318 10835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "0a414f19f41449c8be5976f2b60d33ad"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "35b96545dccc9e4462b1e68020761aba"
server_key_iv: "bf0725c1b52def164e8a0f908145a8c2"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.597772 10835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:34:57.598596 10849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.598783 10835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:57.598874 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "0a414f19f41449c8be5976f2b60d33ad"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "35b96545dccc9e4462b1e68020761aba"
server_key_iv: "bf0725c1b52def164e8a0f908145a8c2"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.598964 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:57.625205 10835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.625592 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.625764 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.626037 10835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:57.626399 10835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:57.626456 10835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.626497 10835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:57.626539 10835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.632616 10835 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:46407
I20260928 06:34:57.632666 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:46407 every 8 connection(s)
I20260928 06:34:57.633010 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:34:57.634605 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10835
I20260928 06:34:57.634742 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:57.634929 24622 external_mini_cluster.cc:1428] Setting key 1f934f6ff6e6b46e489bccaa0a5c3090
I20260928 06:34:57.635887 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--builtin_ntp_servers=127.24.11.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:34:57.639452 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.633738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:51913 (local address 127.24.11.190:38171)
0928 06:34:57.633815 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.633817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.634065 (+ 248us) server_negotiation.cc:407] Connection header received
0928 06:34:57.634726 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.634728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.634741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.634802 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.635727 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.636313 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.637732 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.638051 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.638137 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.639075 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.639077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.639077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:57.639083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.639101 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:57.639105 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.639168 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.639301 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.639330 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:34:57.639626 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.633720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38171 (local address 127.24.11.129:51913)
0928 06:34:57.633930 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.633974 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.634561 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.634905 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.634910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.634921 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:57.635643 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.635651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.636405 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.636406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.637639 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.637644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.638142 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.638143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.638272 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.638927 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:57.638928 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:57.638958 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:57.638983 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:57.639203 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:57.639209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:57.639212 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.639279 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.639364 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260928 06:34:57.640133 10963 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38171
I20260928 06:34:57.640228 10963 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:57.640421 10963 heartbeater.cc:511] Master 127.24.11.190:38171 requested a full tablet report, sending...
I20260928 06:34:57.640959 10775 ts_manager.cc:194] Registered new tserver with Master: 0a414f19f41449c8be5976f2b60d33ad (127.24.11.129:46407)
I20260928 06:34:57.641649 10775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.129:51913
W20260928 06:34:57.717875 10966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:57.718068 10966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:57.718091 10966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:57.719283 10966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:57.719323 10966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:57.719348 10966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:57.720678 10966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:57.720924 10966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:57.721220 10966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:57.726290 10971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.726401 10974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.726300 10966 server_base.cc:1034] running on GCE node
W20260928 06:34:57.726303 10972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.726725 10966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:57.726922 10966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:57.728058 10966 hybrid_clock.cc:648] HybridClock initialized: now 1790577297728047 us; error 27 us; skew 500 ppm
I20260928 06:34:57.729275 10966 webserver.cc:460] Webserver started at http://127.24.11.130:36365/ using document root <none> and password file <none>
I20260928 06:34:57.729512 10966 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:57.729632 10966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:57.729794 10966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:57.730865 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "83fd88517b9d433cb1f00064af45e675"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "a062a9feb1da1332c501dad637de0e3f"
server_key_iv: "003db2b74299b2747bec8331eda6ccfb"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.731225 10966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "83fd88517b9d433cb1f00064af45e675"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "a062a9feb1da1332c501dad637de0e3f"
server_key_iv: "003db2b74299b2747bec8331eda6ccfb"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.732508 10966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:57.733270 10980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.733455 10966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:57.733573 10966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "83fd88517b9d433cb1f00064af45e675"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "a062a9feb1da1332c501dad637de0e3f"
server_key_iv: "003db2b74299b2747bec8331eda6ccfb"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.733664 10966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:57.749943 10966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.750279 10966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.750439 10966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.750682 10966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:57.751035 10966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:57.751091 10966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.751137 10966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:57.751163 10966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.757714 10966 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:33243
I20260928 06:34:57.757771 11093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:33243 every 8 connection(s)
I20260928 06:34:57.758117 10966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:34:57.761157 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 10966
I20260928 06:34:57.761271 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:57.761462 24622 external_mini_cluster.cc:1428] Setting key 8a4883d49bf03918ef2bf0fc1df42415
I20260928 06:34:57.767966 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--builtin_ntp_servers=127.24.11.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:34:57.773056 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.766608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:49531 (local address 127.24.11.190:38171)
0928 06:34:57.766656 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.766658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.767518 (+ 860us) server_negotiation.cc:407] Connection header received
0928 06:34:57.768196 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.768197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.768209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.768239 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.769236 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.769802 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.771398 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.771624 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.771683 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.772699 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.772700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.772700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:57.772705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.772749 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:57.772754 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.772859 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.772969 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.772992 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:34:57.773378 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.766364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38171 (local address 127.24.11.130:49531)
0928 06:34:57.767385 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.767433 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.768077 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.768418 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.768422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.768432 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:57.769100 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.769107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.769898 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.769900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.771277 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.771281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.771720 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.771721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.771911 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.772505 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:57.772506 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:57.772552 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:57.772585 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:57.772869 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:57.772876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:57.772879 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.772964 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.773093 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":91,"threads_started":1}
I20260928 06:34:57.774020 11094 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38171
I20260928 06:34:57.774113 11094 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:57.774324 11094 heartbeater.cc:511] Master 127.24.11.190:38171 requested a full tablet report, sending...
I20260928 06:34:57.774757 10775 ts_manager.cc:194] Registered new tserver with Master: 83fd88517b9d433cb1f00064af45e675 (127.24.11.130:33243)
I20260928 06:34:57.775323 10775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.130:49531
W20260928 06:34:57.851588 11097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:57.851805 11097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:57.851828 11097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:57.853058 11097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:57.853099 11097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:57.853125 11097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:57.854543 11097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:38171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:57.854813 11097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:57.855122 11097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:57.860157 11103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:57.860157 11102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.860230 11097 server_base.cc:1034] running on GCE node
W20260928 06:34:57.860275 11105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:57.860598 11097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:57.860889 11097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:57.862046 11097 hybrid_clock.cc:648] HybridClock initialized: now 1790577297862043 us; error 37 us; skew 500 ppm
I20260928 06:34:57.863430 11097 webserver.cc:460] Webserver started at http://127.24.11.131:46595/ using document root <none> and password file <none>
I20260928 06:34:57.863677 11097 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:57.863724 11097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:57.863931 11097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:57.865002 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "2d90f5bccf884f74ab15af7d7b73360a"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "7f7a7ab45900fac7171f06187c74500b"
server_key_iv: "1f731301a8ea66557fff29b667db7d6a"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.865367 11097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "2d90f5bccf884f74ab15af7d7b73360a"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "7f7a7ab45900fac7171f06187c74500b"
server_key_iv: "1f731301a8ea66557fff29b667db7d6a"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.866696 11097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:57.867413 11111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.867599 11097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:57.867671 11097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "2d90f5bccf884f74ab15af7d7b73360a"
format_stamp: "Formatted at 2026-09-28 06:34:57 on dist-test-slave-38pg"
server_key: "7f7a7ab45900fac7171f06187c74500b"
server_key_iv: "1f731301a8ea66557fff29b667db7d6a"
server_key_version: "encryptionkey@0"
I20260928 06:34:57.867784 11097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:57.891304 11097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:57.891626 11097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:57.891793 11097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:57.892031 11097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:57.892379 11097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:57.892443 11097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.892484 11097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:57.892513 11097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:57.898324 11097 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:34159
I20260928 06:34:57.898376 11224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:34159 every 8 connection(s)
I20260928 06:34:57.898700 11097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:34:57.903492 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11097
I20260928 06:34:57.903601 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:34:57.903837 24622 external_mini_cluster.cc:1428] Setting key 5550509e732ad0ed3d352c32565e7a21
I20260928 06:34:57.903923 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.899401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:42113 (local address 127.24.11.190:38171)
0928 06:34:57.899471 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.899473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.899683 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:34:57.900206 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.900207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.900218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.900286 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.900969 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.901456 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.902499 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.902683 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.902758 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.903635 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.903637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.903637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:57.903643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.903660 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:57.903664 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.903719 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.903806 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.903833 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:34:57.904136 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.899363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:38171 (local address 127.24.11.131:42113)
0928 06:34:57.899548 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.899589 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.900070 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.900362 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.900366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.900374 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:57.900884 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.900888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.901544 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.901545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.902416 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.902418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.902770 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.902771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.902889 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.903470 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:34:57.903470 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:57.903498 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:34:57.903519 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:57.903727 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:57.903732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:57.903734 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.903785 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.903873 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260928 06:34:57.904594 11225 heartbeater.cc:347] Connected to a master server at 127.24.11.190:38171
I20260928 06:34:57.904711 11225 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:57.904917 11225 heartbeater.cc:511] Master 127.24.11.190:38171 requested a full tablet report, sending...
I20260928 06:34:57.905300 10774 ts_manager.cc:194] Registered new tserver with Master: 2d90f5bccf884f74ab15af7d7b73360a (127.24.11.131:34159)
I20260928 06:34:57.905717 10774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.131:42113
I20260928 06:34:57.915830 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:34:57.920197 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.917362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.24.11.190:38171)
0928 06:34:57.917426 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.917428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.917554 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:34:57.917635 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.917635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.917645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.917693 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:57.917842 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.918318 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.919047 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.919221 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.919270 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.919996 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:57.919997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:57.919997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:34:57.920001 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:57.920013 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:34:57.920016 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:57.920086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.920128 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.920137 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:34:57.922096 10774 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:34:57.923641 10774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:34:57.929708 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.925848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:46407 (local address 127.0.0.1:33178)
0928 06:34:57.925965 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.926005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.926064 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.926326 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.926329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.926331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.926398 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.926400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.927494 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.927495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.928374 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.928375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.929378 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.929379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.929454 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.929470 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.929580 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.929631 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260928 06:34:57.929723 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.925957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.24.11.129:46407)
0928 06:34:57.926144 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.926149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.926160 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:34:57.926204 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.926206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.926218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.926324 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.926463 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.927407 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.928522 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.929293 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.929372 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.929470 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.929597 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.929638 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":118,"threads_started":1}
I20260928 06:34:57.930505 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.926062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33243 (local address 127.0.0.1:38820)
0928 06:34:57.926188 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.926196 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.926234 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.926910 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.926911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.926913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.926963 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.926965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.928253 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.928253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.929154 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.929156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.930289 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.930290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.930348 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.930352 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.930397 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.930436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260928 06:34:57.930516 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.926451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.24.11.130:33243)
0928 06:34:57.926684 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.926689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.926699 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:34:57.926737 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.926740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.926754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.927040 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.927096 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.928170 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.929246 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.930190 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.930285 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.930379 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.930407 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.930439 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260928 06:34:57.930552 10897 tablet_service.cc:1467] Processing CreateTablet for tablet 737ccd7724fe4c19b368e78500162796 (DEFAULT_TABLE table=test-table [id=5388005d735a49309d7aa3fd31b2f0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.930979 10897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737ccd7724fe4c19b368e78500162796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.931066 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.926242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:34159 (local address 127.0.0.1:44628)
0928 06:34:57.926617 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:57.926629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:57.926679 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:57.926883 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:57.926884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:57.926886 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:57.926925 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.926929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.928220 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.928220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.929390 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:57.929393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.930828 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.930829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:57.930888 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.930893 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:34:57.930946 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:34:57.930983 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":38,"thread_start_us":82,"threads_started":1}
I20260928 06:34:57.931340 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:57.926372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.24.11.131:34159)
0928 06:34:57.926554 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:57.926558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:57.926744 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:34:57.926792 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:57.926794 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:57.926809 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:57.926883 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:57.926994 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.928116 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.929479 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:57.930416 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:57.930487 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:57.930569 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:57.931209 (+ 640us) server_negotiation.cc:299] Negotiation successful
0928 06:34:57.931249 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260928 06:34:57.931317 11028 tablet_service.cc:1467] Processing CreateTablet for tablet 737ccd7724fe4c19b368e78500162796 (DEFAULT_TABLE table=test-table [id=5388005d735a49309d7aa3fd31b2f0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.931646 11028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737ccd7724fe4c19b368e78500162796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.931921 11159 tablet_service.cc:1467] Processing CreateTablet for tablet 737ccd7724fe4c19b368e78500162796 (DEFAULT_TABLE table=test-table [id=5388005d735a49309d7aa3fd31b2f0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:34:57.932243 11159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 737ccd7724fe4c19b368e78500162796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:57.934347 11239 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Bootstrap starting.
I20260928 06:34:57.934792 11240 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Bootstrap starting.
I20260928 06:34:57.935150 11239 tablet_bootstrap.cc:654] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.935456 11239 log.cc:824] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.935503 11240 tablet_bootstrap.cc:654] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.935704 11241 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Bootstrap starting.
I20260928 06:34:57.935823 11240 log.cc:824] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.936245 11241 tablet_bootstrap.cc:654] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:57.936586 11241 log.cc:824] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:57.936723 11240 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: No bootstrap required, opened a new log
I20260928 06:34:57.936800 11240 ts_tablet_manager.cc:1397] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:57.937331 11241 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: No bootstrap required, opened a new log
I20260928 06:34:57.937407 11241 ts_tablet_manager.cc:1397] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:34:57.939551 11240 raft_consensus.cc:348] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.939741 11240 raft_consensus.cc:374] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.939777 11240 raft_consensus.cc:729] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a414f19f41449c8be5976f2b60d33ad, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.939909 11240 consensus_queue.cc:260] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.939954 11241 raft_consensus.cc:348] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.940114 11241 raft_consensus.cc:374] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.940157 11241 raft_consensus.cc:729] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d90f5bccf884f74ab15af7d7b73360a, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.940177 11240 ts_tablet_manager.cc:1428] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:34:57.940253 10963 heartbeater.cc:503] Master 127.24.11.190:38171 was elected leader, sending a full tablet report...
I20260928 06:34:57.940271 11241 consensus_queue.cc:260] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.940546 11241 ts_tablet_manager.cc:1428] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:34:57.940600 11225 heartbeater.cc:503] Master 127.24.11.190:38171 was elected leader, sending a full tablet report...
I20260928 06:34:57.940893 11239 tablet_bootstrap.cc:492] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: No bootstrap required, opened a new log
I20260928 06:34:57.940959 11239 ts_tablet_manager.cc:1397] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 06:34:57.942934 11239 raft_consensus.cc:348] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.943059 11239 raft_consensus.cc:374] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:57.943080 11239 raft_consensus.cc:729] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fd88517b9d433cb1f00064af45e675, State: Initialized, Role: FOLLOWER
I20260928 06:34:57.943198 11239 consensus_queue.cc:260] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:57.943455 11239 ts_tablet_manager.cc:1428] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:34:57.943528 11094 heartbeater.cc:503] Master 127.24.11.190:38171 was elected leader, sending a full tablet report...
W20260928 06:34:58.008764 11095 tablet.cc:2369] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:34:58.133738 10964 tablet.cc:2369] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:58.145115 11246 raft_consensus.cc:482] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:34:58.145243 11246 raft_consensus.cc:504] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:58.145612 11246 leader_election.cc:288] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a414f19f41449c8be5976f2b60d33ad (127.24.11.129:46407), 83fd88517b9d433cb1f00064af45e675 (127.24.11.130:33243)
W20260928 06:34:58.149220 11226 tablet.cc:2369] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:34:58.149587 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.145975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:46407 (local address 127.24.11.131:52605)
0928 06:34:58.146038 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:58.146052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:58.146105 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:58.146280 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:58.146282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:58.146284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:58.146364 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:58.146366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.147299 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.147300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:58.148559 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:58.148562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.149287 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.149288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:58.149358 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:58.149362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:58.149418 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:34:58.149467 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 06:34:58.149595 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.146057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52605 (local address 127.24.11.129:46407)
0928 06:34:58.146119 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:58.146121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:58.146135 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:34:58.146171 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:58.146172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:58.146185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:58.146266 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:58.146446 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.147213 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.148662 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.149202 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.149278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:58.149286 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:58.149453 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:34:58.149500 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:34:58.149703 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.146216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:52381 (local address 127.24.11.130:33243)
0928 06:34:58.146279 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:58.146281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:58.146395 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:34:58.146444 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:58.146445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:58.146456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:58.146516 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:34:58.146656 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.147322 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.148655 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.149202 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.149278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:58.149286 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:58.149584 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 06:34:58.149640 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:34:58.149714 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.146137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33243 (local address 127.24.11.131:52381)
0928 06:34:58.146322 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:58.146332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:58.146391 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:58.146525 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:58.146525 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:58.146527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:34:58.146590 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:58.146591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.147418 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.147419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:58.148559 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:58.148562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.149357 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.149358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:58.149444 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:58.149448 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:34:58.149551 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 06:34:58.149588 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260928 06:34:58.150043 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737ccd7724fe4c19b368e78500162796" candidate_uuid: "2d90f5bccf884f74ab15af7d7b73360a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a414f19f41449c8be5976f2b60d33ad" is_pre_election: true
I20260928 06:34:58.150058 11048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737ccd7724fe4c19b368e78500162796" candidate_uuid: "2d90f5bccf884f74ab15af7d7b73360a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fd88517b9d433cb1f00064af45e675" is_pre_election: true
I20260928 06:34:58.150218 10917 raft_consensus.cc:2454] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90f5bccf884f74ab15af7d7b73360a in term 0.
I20260928 06:34:58.150238 11048 raft_consensus.cc:2454] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d90f5bccf884f74ab15af7d7b73360a in term 0.
I20260928 06:34:58.150462 11114 leader_election.cc:302] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d90f5bccf884f74ab15af7d7b73360a, 83fd88517b9d433cb1f00064af45e675; no voters:
I20260928 06:34:58.150589 11246 raft_consensus.cc:2781] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:34:58.150683 11246 raft_consensus.cc:482] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:34:58.150722 11246 raft_consensus.cc:3037] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:58.151448 11246 raft_consensus.cc:504] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:58.151595 11246 leader_election.cc:288] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [CANDIDATE]: Term 1 election: Requested vote from peers 0a414f19f41449c8be5976f2b60d33ad (127.24.11.129:46407), 83fd88517b9d433cb1f00064af45e675 (127.24.11.130:33243)
I20260928 06:34:58.151837 10917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737ccd7724fe4c19b368e78500162796" candidate_uuid: "2d90f5bccf884f74ab15af7d7b73360a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a414f19f41449c8be5976f2b60d33ad"
I20260928 06:34:58.151851 11048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737ccd7724fe4c19b368e78500162796" candidate_uuid: "2d90f5bccf884f74ab15af7d7b73360a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fd88517b9d433cb1f00064af45e675"
I20260928 06:34:58.151926 10917 raft_consensus.cc:3037] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:58.151926 11048 raft_consensus.cc:3037] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:58.152864 11048 raft_consensus.cc:2454] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90f5bccf884f74ab15af7d7b73360a in term 1.
I20260928 06:34:58.152875 10917 raft_consensus.cc:2454] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d90f5bccf884f74ab15af7d7b73360a in term 1.
I20260928 06:34:58.153079 11114 leader_election.cc:302] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d90f5bccf884f74ab15af7d7b73360a, 83fd88517b9d433cb1f00064af45e675; no voters:
I20260928 06:34:58.153167 11246 raft_consensus.cc:2781] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:58.153327 11246 raft_consensus.cc:686] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 1 LEADER]: Becoming Leader. State: Replica: 2d90f5bccf884f74ab15af7d7b73360a, State: Running, Role: LEADER
I20260928 06:34:58.153434 11246 consensus_queue.cc:237] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } }
I20260928 06:34:58.154381 10775 catalog_manager.cc:5495] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d90f5bccf884f74ab15af7d7b73360a (127.24.11.131). New cstate: current_term: 1 leader_uuid: "2d90f5bccf884f74ab15af7d7b73360a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d90f5bccf884f74ab15af7d7b73360a" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 34159 } health_report { overall_health: HEALTHY } } }
I20260928 06:34:58.161039 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.158557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.24.11.131:34159)
0928 06:34:58.158611 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:58.158613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:58.158622 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:34:58.158711 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:58.158711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:58.158729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:58.158837 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:34:58.158925 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.159383 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.160207 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:58.160403 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:58.160484 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:58.160643 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:34:58.160738 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:34:58.160882 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:58.160932 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:34:58.160940 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:34:58.163028 10917 raft_consensus.cc:1261] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 1 FOLLOWER]: Refusing update from remote peer 2d90f5bccf884f74ab15af7d7b73360a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:58.163018 11048 raft_consensus.cc:1261] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 1 FOLLOWER]: Refusing update from remote peer 2d90f5bccf884f74ab15af7d7b73360a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:34:58.163414 11246 consensus_queue.cc:1035] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fd88517b9d433cb1f00064af45e675" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:58.163534 11249 consensus_queue.cc:1035] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a414f19f41449c8be5976f2b60d33ad" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:34:58.166070 11253 mvcc.cc:205] Tried to move back new op lower bound from 7334204613272801280 to 7334204613237174272. Current Snapshot: MvccSnapshot[applied={T|T < 7334204613272801280}]
I20260928 06:34:58.166822 11254 mvcc.cc:205] Tried to move back new op lower bound from 7334204613272801280 to 7334204613237174272. Current Snapshot: MvccSnapshot[applied={T|T < 7334204613272801280 or (T in {7334204613272801280})}]
I20260928 06:34:58.168221 10775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:58.168300 10775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:34:58.169088 10775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abf82d2d739048b6b75291155db33a12: Sending DeleteTablet for 3 replicas of tablet 737ccd7724fe4c19b368e78500162796
I20260928 06:34:58.169395 10897 tablet_service.cc:1514] Processing DeleteTablet for tablet 737ccd7724fe4c19b368e78500162796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:58 UTC) from {username='slave'} at 127.0.0.1:33178
I20260928 06:34:58.169456 11028 tablet_service.cc:1514] Processing DeleteTablet for tablet 737ccd7724fe4c19b368e78500162796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:58 UTC) from {username='slave'} at 127.0.0.1:38820
I20260928 06:34:58.169500 11159 tablet_service.cc:1514] Processing DeleteTablet for tablet 737ccd7724fe4c19b368e78500162796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:34:58 UTC) from {username='slave'} at 127.0.0.1:44628
I20260928 06:34:58.169796 11263 tablet_replica.cc:330] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: stopping tablet replica
I20260928 06:34:58.169869 11264 tablet_replica.cc:330] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad: stopping tablet replica
I20260928 06:34:58.169963 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10835
I20260928 06:34:58.169987 11264 raft_consensus.cc:2229] T 737ccd7724fe4c19b368e78500162796 P 0a414f19f41449c8be5976f2b60d33ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:58.169983 11263 raft_consensus.cc:2229] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:34:58.171885 11262 tablet_replica.cc:330] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: stopping tablet replica
I20260928 06:34:58.172010 11262 raft_consensus.cc:2229] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:34:58.172124 11262 raft_consensus.cc:2258] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:58.172535 11262 ts_tablet_manager.cc:1905] T 737ccd7724fe4c19b368e78500162796 P 83fd88517b9d433cb1f00064af45e675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:58.173703 11263 raft_consensus.cc:2258] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:34:58.174422 11263 ts_tablet_manager.cc:1905] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:34:58.175745 11263 ts_tablet_manager.cc:1918] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:34:58.175832 11263 log.cc:1196] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/737ccd7724fe4c19b368e78500162796
I20260928 06:34:58.176123 11263 ts_tablet_manager.cc:1939] T 737ccd7724fe4c19b368e78500162796 P 2d90f5bccf884f74ab15af7d7b73360a: Deleting consensus metadata
W20260928 06:34:58.176223 10761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:46407 (error 108)
I20260928 06:34:58.176515 10761 catalog_manager.cc:4841] TS 2d90f5bccf884f74ab15af7d7b73360a (127.24.11.131:34159): tablet 737ccd7724fe4c19b368e78500162796 (table test-table [id=5388005d735a49309d7aa3fd31b2f0a3]) successfully deleted
I20260928 06:34:58.176530 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10966
I20260928 06:34:58.177132 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.176671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:46407 (local address 127.0.0.1:33182)
0928 06:34:58.177056 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:58.177093 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:46407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365}
W20260928 06:34:58.177228 10761 catalog_manager.cc:4584] TS 0a414f19f41449c8be5976f2b60d33ad (127.24.11.129:46407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 737ccd7724fe4c19b368e78500162796: Network error: Client connection negotiation failed: client connection to 127.24.11.129:46407: connect: Connection refused (error 111)
I20260928 06:34:58.182120 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:58.181963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:33243 (local address 127.0.0.1:38836)
0928 06:34:58.182029 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:58.182060 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.130:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:34:58.182143 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11097
I20260928 06:34:58.186968 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 10744
2026-09-28T06:34:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:34:58 dist-test-slave-38pg krb5kdc[11268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:34:58 dist-test-slave-38pg krb5kdc[11268](info): set up 2 sockets
Sep 28 06:34:58 dist-test-slave-38pg krb5kdc[11268](info): commencing operation
krb5kdc: starting...
W20260928 06:34:59.235140 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:34:59.360414 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44193
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:44193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:59.442265 11281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:59.442533 11281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:59.442574 11281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:59.443751 11281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:34:59.443822 11281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:59.443846 11281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:59.443866 11281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:34:59.443893 11281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:34:59.445439 11281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:44193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:59.445750 11281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:59.445989 11281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:59.450989 11286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.450970 11287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.450969 11289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.451212 11281 server_base.cc:1034] running on GCE node
I20260928 06:34:59.451455 11281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:59.451771 11281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:59.452924 11281 hybrid_clock.cc:648] HybridClock initialized: now 1790577299452908 us; error 45 us; skew 500 ppm
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:59.460855 11281 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:34:59.461410 11281 webserver.cc:460] Webserver started at http://127.24.11.190:44729/ using document root <none> and password file <none>
I20260928 06:34:59.461719 11281 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:59.461791 11281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:59.461961 11281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:59.463063 11281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "d185fef5f2ea427dbf08b5b29b571f65"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "82a0235659a4c6eed61f5b11c1991f07"
server_key_iv: "f9cbd8ae1c7738c5ff5665427027a974"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.463457 11281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "d185fef5f2ea427dbf08b5b29b571f65"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "82a0235659a4c6eed61f5b11c1991f07"
server_key_iv: "f9cbd8ae1c7738c5ff5665427027a974"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.464984 11281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:59.465986 11296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.466207 11281 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:59.466331 11281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "d185fef5f2ea427dbf08b5b29b571f65"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "82a0235659a4c6eed61f5b11c1991f07"
server_key_iv: "f9cbd8ae1c7738c5ff5665427027a974"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.466416 11281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:59.480821 11281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:59.483191 11281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:59.483376 11281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:59.487931 11281 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:44193
I20260928 06:34:59.487980 11348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:44193 every 8 connection(s)
I20260928 06:34:59.488343 11281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:34:59.489228 11349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:34:59.491604 11349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: Bootstrap starting.
I20260928 06:34:59.492152 11349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: Neither blocks nor log segments found. Creating new log.
I20260928 06:34:59.492385 11349 log.cc:824] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: Log is configured to *not* fsync() on all Append() calls
I20260928 06:34:59.493053 11349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: No bootstrap required, opened a new log
I20260928 06:34:59.494787 11349 raft_consensus.cc:348] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } }
I20260928 06:34:59.494935 11349 raft_consensus.cc:374] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:34:59.494959 11349 raft_consensus.cc:729] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d185fef5f2ea427dbf08b5b29b571f65, State: Initialized, Role: FOLLOWER
I20260928 06:34:59.495069 11349 consensus_queue.cc:260] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } }
I20260928 06:34:59.495121 11349 raft_consensus.cc:388] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:34:59.495142 11349 raft_consensus.cc:482] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:34:59.495229 11349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:34:59.495555 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11281
I20260928 06:34:59.495675 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:34:59.495878 24622 external_mini_cluster.cc:1428] Setting key a88a097c738eecc4fc35713bebb3352d
I20260928 06:34:59.495872 11349 raft_consensus.cc:504] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } }
I20260928 06:34:59.495977 11349 leader_election.cc:302] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d185fef5f2ea427dbf08b5b29b571f65; no voters:
I20260928 06:34:59.496184 11349 leader_election.cc:288] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:34:59.496209 11352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:34:59.496407 11352 raft_consensus.cc:686] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [term 1 LEADER]: Becoming Leader. State: Replica: d185fef5f2ea427dbf08b5b29b571f65, State: Running, Role: LEADER
I20260928 06:34:59.496479 11349 sys_catalog.cc:564] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:34:59.496505 11352 consensus_queue.cc:237] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } }
I20260928 06:34:59.496929 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d185fef5f2ea427dbf08b5b29b571f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } } }
I20260928 06:34:59.497016 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:59.497164 11354 sys_catalog.cc:455] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d185fef5f2ea427dbf08b5b29b571f65. Latest consensus state: current_term: 1 leader_uuid: "d185fef5f2ea427dbf08b5b29b571f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d185fef5f2ea427dbf08b5b29b571f65" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 44193 } } }
I20260928 06:34:59.497239 11362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:34:59.497397 11354 sys_catalog.cc:458] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65 [sys.catalog]: This master's current role is: LEADER
I20260928 06:34:59.498207 11362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:34:59.499716 11362 catalog_manager.cc:1347] Generated new cluster ID: 68526918371441eeab280a86ec344bd9
I20260928 06:34:59.499768 11362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:59.506242 11362 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:34:59.506989 11362 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:34:59.509472 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:59.496622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.24.11.190:44193)
0928 06:34:59.497064 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:59.497076 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:59.497114 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:34:59.497264 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:59.497275 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:59.497334 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:59.497471 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:59.498557 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.499944 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.501005 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.501714 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.501827 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:59.504761 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:59.504766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:59.504766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:59.504780 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:59.507695 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.508157 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.508161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.508161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.508273 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.508500 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.508501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.508501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.508788 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:59.508952 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:59.509086 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:34:59.509246 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":97,"threads_started":1}
I20260928 06:34:59.521507 11362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: Generated new TSK 0
I20260928 06:34:59.521768 11362 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.11.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
Entry for principal kudu/127.24.11.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab.
I20260928 06:34:59.552987 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:59.646871 11375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:59.647094 11375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:59.647169 11375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:59.648399 11375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:59.648444 11375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:59.648474 11375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:34:59.650060 11375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.129.keytab
--principal=kudu/127.24.11.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:59.650321 11375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:59.650566 11375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:59.656745 11380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.656740 11381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.656903 11375 server_base.cc:1034] running on GCE node
W20260928 06:34:59.656729 11383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.657186 11375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:59.657507 11375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:59.658752 11375 hybrid_clock.cc:648] HybridClock initialized: now 1790577299658746 us; error 35 us; skew 500 ppm
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:59.661492 11375 init.cc:377] Logged in from keytab as kudu/127.24.11.129@KRBTEST.COM (short username kudu)
I20260928 06:34:59.662135 11375 webserver.cc:460] Webserver started at http://127.24.11.129:42417/ using document root <none> and password file <none>
I20260928 06:34:59.662395 11375 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:59.662472 11375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:59.662633 11375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:59.663832 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "b0f0941f30504963adf42edd0e42bb66"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "dc1174d455181f5c18a0c4e350252938"
server_key_iv: "9ff8e981034e492c545870bbee1fdde7"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.664243 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "b0f0941f30504963adf42edd0e42bb66"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "dc1174d455181f5c18a0c4e350252938"
server_key_iv: "9ff8e981034e492c545870bbee1fdde7"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.665944 11375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:34:59.666733 11390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.666923 11375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:34:59.667029 11375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "b0f0941f30504963adf42edd0e42bb66"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "dc1174d455181f5c18a0c4e350252938"
server_key_iv: "9ff8e981034e492c545870bbee1fdde7"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.667110 11375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:59.690066 11375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:59.692220 11375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:59.692405 11375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:59.692660 11375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:59.692993 11375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:59.693048 11375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.693095 11375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:59.693120 11375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.699465 11375 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:43823
I20260928 06:34:59.699509 11503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:43823 every 8 connection(s)
I20260928 06:34:59.699868 11375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.129@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:59.709075 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11375
I20260928 06:34:59.709210 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:34:59.709429 24622 external_mini_cluster.cc:1428] Setting key f63b5efe7f323576328aeec97a0f0312
I20260928 06:34:59.709533 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:59.700649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:44061 (local address 127.24.11.190:44193)
0928 06:34:59.700705 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:59.700707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:59.700947 (+ 240us) server_negotiation.cc:407] Connection header received
0928 06:34:59.701430 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:59.701431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:59.701441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:59.701497 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:59.702481 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.703137 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.704121 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.704312 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.704363 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:59.706625 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:59.706626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:59.706626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:59.706634 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:59.708036 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.708414 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.708414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.708415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.708467 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.708677 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.708677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.708678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.708800 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:59.708854 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:59.709342 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 06:34:59.709424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:34:59.709775 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:59.700645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44193 (local address 127.24.11.129:44061)
0928 06:34:59.700819 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:59.700858 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:59.701347 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:59.701577 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:59.701581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:59.702170 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:59.702394 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:59.702399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.703224 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.703225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:59.704046 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:59.704048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.704383 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.704384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:59.704484 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:59.705096 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:59.705104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:59.706525 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:59.708114 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:59.708121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:59.708122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:59.708350 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:59.708525 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:59.708526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:59.708526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:59.708625 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:59.708910 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:59.708916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:59.709099 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 06:34:59.709263 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 06:34:59.709478 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260928 06:34:59.710443 11504 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44193
I20260928 06:34:59.710562 11504 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:59.710748 11504 heartbeater.cc:511] Master 127.24.11.190:44193 requested a full tablet report, sending...
I20260928 06:34:59.711319 11313 ts_manager.cc:194] Registered new tserver with Master: b0f0941f30504963adf42edd0e42bb66 (127.24.11.129:43823)
I20260928 06:34:59.712054 11313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.129@KRBTEST.COM'} at 127.24.11.129:44061
WARNING: no policy specified for kudu/127.24.11.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
Entry for principal kudu/127.24.11.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab.
I20260928 06:34:59.736475 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:59.822338 11509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:59.822584 11509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:59.822623 11509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:59.823783 11509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:59.823858 11509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:59.823901 11509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:34:59.825398 11509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.130.keytab
--principal=kudu/127.24.11.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:59.825691 11509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:59.825942 11509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:34:59.831449 11514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.831501 11509 server_base.cc:1034] running on GCE node
W20260928 06:34:59.831456 11517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.831455 11515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.831907 11509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:59.832131 11509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:59.833277 11509 hybrid_clock.cc:648] HybridClock initialized: now 1790577299833266 us; error 35 us; skew 500 ppm
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:59.836163 11509 init.cc:377] Logged in from keytab as kudu/127.24.11.130@KRBTEST.COM (short username kudu)
I20260928 06:34:59.836650 11509 webserver.cc:460] Webserver started at http://127.24.11.130:44033/ using document root <none> and password file <none>
I20260928 06:34:59.836920 11509 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:59.836992 11509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:59.837141 11509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:59.838303 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "82c824706cb340509b84d079c8ae16d6"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "24a5cfb73c6438b554a09825cd93887f"
server_key_iv: "1fd95d6eb1240ac29ab8d34cd87595d4"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.838675 11509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "82c824706cb340509b84d079c8ae16d6"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "24a5cfb73c6438b554a09825cd93887f"
server_key_iv: "1fd95d6eb1240ac29ab8d34cd87595d4"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.840077 11509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:34:59.840901 11524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.841102 11509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:34:59.841204 11509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "82c824706cb340509b84d079c8ae16d6"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "24a5cfb73c6438b554a09825cd93887f"
server_key_iv: "1fd95d6eb1240ac29ab8d34cd87595d4"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.841291 11509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:34:59.852883 11509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:34:59.855382 11509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:34:59.855502 11509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:34:59.855703 11509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:34:59.856035 11509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:34:59.856069 11509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.856114 11509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:34:59.856141 11509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:34:59.862306 11509 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:40481
I20260928 06:34:59.862380 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:40481 every 8 connection(s)
I20260928 06:34:59.862692 11509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.130@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:34:59.871918 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11509
I20260928 06:34:59.872052 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:34:59.872289 24622 external_mini_cluster.cc:1428] Setting key 0e8fe59d164e129f7e8ab20fe7b9a255
I20260928 06:34:59.873049 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:59.863589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:34549 (local address 127.24.11.190:44193)
0928 06:34:59.863648 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:34:59.863651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:34:59.863819 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:34:59.864386 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:34:59.864388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:34:59.864401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:34:59.864456 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:34:59.865496 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.866072 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.867139 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.867340 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.867416 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:59.869867 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:34:59.869869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:34:59.869870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:34:59.869877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:34:59.871358 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.871846 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.871847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.871847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.871918 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:34:59.872261 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:34:59.872262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:34:59.872262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:34:59.872442 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:34:59.872528 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:34:59.872857 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 06:34:59.872946 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:34:59.873250 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:34:59.863484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44193 (local address 127.24.11.130:34549)
0928 06:34:59.863668 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:34:59.863718 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:34:59.864275 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:34:59.864544 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:34:59.864548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:34:59.865168 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:34:59.865382 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:59.865387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.866184 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.866185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:59.867030 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:34:59.867032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:34:59.867437 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:34:59.867438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:34:59.867570 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:34:59.868162 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:34:59.868171 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:34:59.869767 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:34:59.871466 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:59.871475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:59.871477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:59.871758 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:59.872001 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:34:59.872002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:34:59.872003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:34:59.872162 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:34:59.872523 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:34:59.872529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:34:59.872719 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 06:34:59.872813 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:34:59.872988 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260928 06:34:59.873781 11638 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44193
I20260928 06:34:59.873900 11638 heartbeater.cc:464] Registering TS with master...
I20260928 06:34:59.874096 11638 heartbeater.cc:511] Master 127.24.11.190:44193 requested a full tablet report, sending...
I20260928 06:34:59.874543 11313 ts_manager.cc:194] Registered new tserver with Master: 82c824706cb340509b84d079c8ae16d6 (127.24.11.130:40481)
I20260928 06:34:59.875012 11313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.130@KRBTEST.COM'} at 127.24.11.130:34549
WARNING: no policy specified for kudu/127.24.11.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
Entry for principal kudu/127.24.11.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab.
I20260928 06:34:59.899926 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:34:59.982259 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:34:59.982522 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:34:59.982561 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:34:59.983809 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:34:59.983884 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:34:59.983927 11643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:34:59.985576 11643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.131.keytab
--principal=kudu/127.24.11.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:44193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:34:59.985877 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:34:59.986123 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:34:59.991438 11643 server_base.cc:1034] running on GCE node
W20260928 06:34:59.991444 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.991571 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:34:59.991457 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:34:59.991866 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:34:59.992156 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:34:59.993323 11643 hybrid_clock.cc:648] HybridClock initialized: now 1790577299993311 us; error 30 us; skew 500 ppm
Sep 28 06:34:59 dist-test-slave-38pg krb5kdc[11268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:34:59.996045 11643 init.cc:377] Logged in from keytab as kudu/127.24.11.131@KRBTEST.COM (short username kudu)
I20260928 06:34:59.996486 11643 webserver.cc:460] Webserver started at http://127.24.11.131:46051/ using document root <none> and password file <none>
I20260928 06:34:59.996816 11643 fs_manager.cc:359] Metadata directory not provided
I20260928 06:34:59.996872 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:34:59.997000 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:34:59.998229 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "20395c3a27ce4c78bcfa06dca557b5a2"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "1b5255ddb59f1bc7ddf7ed313f4c2b3d"
server_key_iv: "4e2043840d8bf993c08af8c4460475a7"
server_key_version: "encryptionkey@0"
I20260928 06:34:59.998615 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "20395c3a27ce4c78bcfa06dca557b5a2"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "1b5255ddb59f1bc7ddf7ed313f4c2b3d"
server_key_iv: "4e2043840d8bf993c08af8c4460475a7"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.000231 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:00.001048 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.001295 11643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:00.001410 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "20395c3a27ce4c78bcfa06dca557b5a2"
format_stamp: "Formatted at 2026-09-28 06:34:59 on dist-test-slave-38pg"
server_key: "1b5255ddb59f1bc7ddf7ed313f4c2b3d"
server_key_iv: "4e2043840d8bf993c08af8c4460475a7"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.001503 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:00.022562 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:00.024775 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:00.024937 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:00.025188 11643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:00.025571 11643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:00.025620 11643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.025665 11643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:00.025691 11643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.031747 11643 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:45559
I20260928 06:35:00.031790 11771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:45559 every 8 connection(s)
I20260928 06:35:00.032173 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:00.035395 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11643
I20260928 06:35:00.035523 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:00.035724 24622 external_mini_cluster.cc:1428] Setting key 31787ff79fb531edf7ddc71b15660117
Sep 28 06:35:00 dist-test-slave-38pg krb5kdc[11268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577299, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.131@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:00.042253 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.032932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:54861 (local address 127.24.11.190:44193)
0928 06:35:00.032998 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.033000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.033191 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:35:00.033806 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.033807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.033838 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.033895 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.034889 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.035394 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.036671 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.036881 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.036988 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.039368 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.039369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.039370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:00.039377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.040751 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:00.041211 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:00.041212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:00.041213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:00.041276 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:00.041562 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:00.041563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:00.041563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:00.041687 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.041761 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.042065 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.042158 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:35:00.042448 11774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.032917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:44193 (local address 127.24.11.131:54861)
0928 06:35:00.033069 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.033107 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.033697 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.033975 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.033979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.034559 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:00.034774 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.034779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.035493 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.035494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.036575 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.036577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.036999 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.037000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.037135 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.037774 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:00.037783 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:00.039228 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:00.040885 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:00.040892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:00.040894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:00.041124 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:00.041355 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:00.041356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:00.041356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:00.041469 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:00.041765 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:00.041772 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:00.041922 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.042030 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.042193 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 06:35:00.042984 11772 heartbeater.cc:347] Connected to a master server at 127.24.11.190:44193
I20260928 06:35:00.043087 11772 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:00.043251 11772 heartbeater.cc:511] Master 127.24.11.190:44193 requested a full tablet report, sending...
I20260928 06:35:00.043607 11313 ts_manager.cc:194] Registered new tserver with Master: 20395c3a27ce4c78bcfa06dca557b5a2 (127.24.11.131:45559)
I20260928 06:35:00.044004 11313 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.11.131@KRBTEST.COM'} at 127.24.11.131:54861
I20260928 06:35:00.047664 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:00.055433 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.049031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.24.11.190:44193)
0928 06:35:00.049104 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.049106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.049164 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:35:00.049213 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.049214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.049222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.049271 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.049943 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.050420 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.051155 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.051342 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.051392 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.052723 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.052724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.052724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:00.052729 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.054109 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:00.054521 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:00.054522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:00.054523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:00.054584 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:00.054850 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:00.054851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:00.054851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:00.055022 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.055115 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.055210 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.055293 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 06:35:00.057286 11313 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:00.059031 11313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:00.066103 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45559 (local address 127.0.0.1:47370)
0928 06:35:00.062157 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.062165 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.062206 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.062552 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.062553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.062555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.062611 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.062613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063502 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.063503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.064911 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.064914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.065754 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.065851 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.065871 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.065942 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.066009 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":551,"thread_start_us":79,"threads_started":1}
I20260928 06:35:00.066212 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:43823 (local address 127.0.0.1:56406)
0928 06:35:00.062130 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.062140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.062198 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.062562 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.062563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.062565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.062608 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.062609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063925 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.063926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.065013 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.065016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.065902 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.065969 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.065978 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.066034 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.066078 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260928 06:35:00.066191 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.24.11.129:43823)
0928 06:35:00.061828 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.061833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.062400 (+ 567us) server_negotiation.cc:407] Connection header received
0928 06:35:00.062446 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.062449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.062466 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.062678 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.062766 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063822 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065116 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.065786 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065891 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.065955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.066045 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.066086 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260928 06:35:00.066695 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.24.11.131:45559)
0928 06:35:00.062320 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.062325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.062335 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:00.062372 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.062374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.062390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.062464 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.062703 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063383 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065014 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.065669 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065753 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.065815 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.066551 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.066603 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":80,"threads_started":1}
I20260928 06:35:00.066711 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:40481 (local address 127.0.0.1:34828)
0928 06:35:00.061859 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.061899 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.061965 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.062222 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.062226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.062228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.062314 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.062316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063698 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.063698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.064911 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.064914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.066501 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.066502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.066564 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.066569 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.066601 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.066641 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":7,"thread_start_us":175,"threads_started":1}
I20260928 06:35:00.067096 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.061843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.24.11.130:40481)
0928 06:35:00.062015 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.062020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.062033 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:35:00.062081 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.062083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.062101 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.062205 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.062381 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.063585 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.065640 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.066403 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.066879 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.066945 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.066971 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.067015 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260928 06:35:00.067216 11438 tablet_service.cc:1467] Processing CreateTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 (DEFAULT_TABLE table=test-table [id=c0daec46caa345399a6b3948ce5d79e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.067461 11706 tablet_service.cc:1467] Processing CreateTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 (DEFAULT_TABLE table=test-table [id=c0daec46caa345399a6b3948ce5d79e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.067584 11438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05de6a954bca42fea5e9b6b4eacb42d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.067796 11706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05de6a954bca42fea5e9b6b4eacb42d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.067715 11572 tablet_service.cc:1467] Processing CreateTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 (DEFAULT_TABLE table=test-table [id=c0daec46caa345399a6b3948ce5d79e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.068008 11572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05de6a954bca42fea5e9b6b4eacb42d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.070729 11789 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Bootstrap starting.
I20260928 06:35:00.071533 11789 tablet_bootstrap.cc:654] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.071537 11790 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Bootstrap starting.
I20260928 06:35:00.071772 11791 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Bootstrap starting.
I20260928 06:35:00.072260 11790 tablet_bootstrap.cc:654] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.072415 11791 tablet_bootstrap.cc:654] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.072662 11789 log.cc:824] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.072950 11790 log.cc:824] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.073062 11791 log.cc:824] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.073619 11789 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: No bootstrap required, opened a new log
I20260928 06:35:00.073719 11789 ts_tablet_manager.cc:1397] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:35:00.073820 11790 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: No bootstrap required, opened a new log
I20260928 06:35:00.073853 11791 tablet_bootstrap.cc:492] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: No bootstrap required, opened a new log
I20260928 06:35:00.073908 11790 ts_tablet_manager.cc:1397] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:00.073908 11791 ts_tablet_manager.cc:1397] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:00.075245 11789 raft_consensus.cc:348] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.075423 11789 raft_consensus.cc:374] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.075469 11789 raft_consensus.cc:729] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f0941f30504963adf42edd0e42bb66, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.075623 11789 consensus_queue.cc:260] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.075940 11504 heartbeater.cc:503] Master 127.24.11.190:44193 was elected leader, sending a full tablet report...
I20260928 06:35:00.075877 11790 raft_consensus.cc:348] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.076057 11790 raft_consensus.cc:374] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.076097 11790 raft_consensus.cc:729] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c824706cb340509b84d079c8ae16d6, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.076156 11789 ts_tablet_manager.cc:1428] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:35:00.076123 11791 raft_consensus.cc:348] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.076210 11790 consensus_queue.cc:260] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.076267 11791 raft_consensus.cc:374] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.076299 11791 raft_consensus.cc:729] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20395c3a27ce4c78bcfa06dca557b5a2, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.076418 11791 consensus_queue.cc:260] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.076476 11790 ts_tablet_manager.cc:1428] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:00.076617 11638 heartbeater.cc:503] Master 127.24.11.190:44193 was elected leader, sending a full tablet report...
I20260928 06:35:00.076692 11791 ts_tablet_manager.cc:1428] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:35:00.076818 11772 heartbeater.cc:503] Master 127.24.11.190:44193 was elected leader, sending a full tablet report...
W20260928 06:35:00.113313 11639 tablet.cc:2369] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:00.173954 11796 raft_consensus.cc:482] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:00.174088 11796 raft_consensus.cc:504] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.174533 11796 leader_election.cc:288] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20395c3a27ce4c78bcfa06dca557b5a2 (127.24.11.131:45559), b0f0941f30504963adf42edd0e42bb66 (127.24.11.129:43823)
I20260928 06:35:00.178603 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.174905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:44705 (local address 127.24.11.129:43823)
0928 06:35:00.174964 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.174966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.175119 (+ 153us) server_negotiation.cc:407] Connection header received
0928 06:35:00.175156 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.175157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.175164 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.175231 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.175365 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.176056 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.177637 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.178217 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.178298 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.178307 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.178488 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.178530 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:35:00.178603 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.174875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:35723 (local address 127.24.11.131:45559)
0928 06:35:00.174932 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.174934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.174945 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:00.174980 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.174981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.174991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.175054 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.175233 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.176025 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.177637 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.178215 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.178298 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.178307 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.178492 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.178530 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:35:00.178620 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.174780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:45559 (local address 127.24.11.130:35723)
0928 06:35:00.174843 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.174858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.174910 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.175079 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.175080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.175082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.175156 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.175159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.176144 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.176145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.177514 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.177517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.178314 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.178315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.178403 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.178405 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.178467 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.178521 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":59}
I20260928 06:35:00.178627 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.174798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:43823 (local address 127.24.11.130:44705)
0928 06:35:00.175023 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.175030 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.175099 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.175239 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.175240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.175241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.175301 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.175302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.176162 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.176163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.177514 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.177517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.178314 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.178315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.178399 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.178403 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.178467 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.178521 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":97,"thread_start_us":125,"threads_started":1}
I20260928 06:35:00.179070 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05de6a954bca42fea5e9b6b4eacb42d0" candidate_uuid: "82c824706cb340509b84d079c8ae16d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" is_pre_election: true
I20260928 06:35:00.179270 11726 raft_consensus.cc:2454] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c824706cb340509b84d079c8ae16d6 in term 0.
I20260928 06:35:00.179312 11458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05de6a954bca42fea5e9b6b4eacb42d0" candidate_uuid: "82c824706cb340509b84d079c8ae16d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0941f30504963adf42edd0e42bb66" is_pre_election: true
I20260928 06:35:00.179471 11458 raft_consensus.cc:2454] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c824706cb340509b84d079c8ae16d6 in term 0.
I20260928 06:35:00.179530 11528 leader_election.cc:302] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20395c3a27ce4c78bcfa06dca557b5a2, 82c824706cb340509b84d079c8ae16d6; no voters:
I20260928 06:35:00.179667 11796 raft_consensus.cc:2781] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:00.179723 11796 raft_consensus.cc:482] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:00.179745 11796 raft_consensus.cc:3037] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.180646 11796 raft_consensus.cc:504] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.180878 11796 leader_election.cc:288] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [CANDIDATE]: Term 1 election: Requested vote from peers 20395c3a27ce4c78bcfa06dca557b5a2 (127.24.11.131:45559), b0f0941f30504963adf42edd0e42bb66 (127.24.11.129:43823)
I20260928 06:35:00.181005 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05de6a954bca42fea5e9b6b4eacb42d0" candidate_uuid: "82c824706cb340509b84d079c8ae16d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20395c3a27ce4c78bcfa06dca557b5a2"
I20260928 06:35:00.181052 11458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05de6a954bca42fea5e9b6b4eacb42d0" candidate_uuid: "82c824706cb340509b84d079c8ae16d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f0941f30504963adf42edd0e42bb66"
I20260928 06:35:00.181105 11726 raft_consensus.cc:3037] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.181151 11458 raft_consensus.cc:3037] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.181978 11458 raft_consensus.cc:2454] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c824706cb340509b84d079c8ae16d6 in term 1.
I20260928 06:35:00.181985 11726 raft_consensus.cc:2454] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c824706cb340509b84d079c8ae16d6 in term 1.
I20260928 06:35:00.182164 11528 leader_election.cc:302] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20395c3a27ce4c78bcfa06dca557b5a2, 82c824706cb340509b84d079c8ae16d6; no voters:
I20260928 06:35:00.182289 11796 raft_consensus.cc:2781] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:00.182466 11796 raft_consensus.cc:686] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 1 LEADER]: Becoming Leader. State: Replica: 82c824706cb340509b84d079c8ae16d6, State: Running, Role: LEADER
I20260928 06:35:00.182571 11796 consensus_queue.cc:237] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } }
I20260928 06:35:00.183419 11313 catalog_manager.cc:5495] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c824706cb340509b84d079c8ae16d6 (127.24.11.130). New cstate: current_term: 1 leader_uuid: "82c824706cb340509b84d079c8ae16d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c824706cb340509b84d079c8ae16d6" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 40481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 } health_report { overall_health: UNKNOWN } } }
W20260928 06:35:00.200628 11505 tablet.cc:2369] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:00.214282 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.211653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.24.11.130:40481)
0928 06:35:00.211738 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.211740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.211750 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:00.211802 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.211803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.211822 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.211892 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:00.212057 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.212557 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.213372 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.213664 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.213729 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.213926 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:00.214037 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:00.214103 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.214134 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.214156 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 06:35:00.216336 11726 raft_consensus.cc:1261] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 1 FOLLOWER]: Refusing update from remote peer 82c824706cb340509b84d079c8ae16d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:00.216337 11458 raft_consensus.cc:1261] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 1 FOLLOWER]: Refusing update from remote peer 82c824706cb340509b84d079c8ae16d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:00.216753 11799 consensus_queue.cc:1035] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20395c3a27ce4c78bcfa06dca557b5a2" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 45559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:00.216930 11796 consensus_queue.cc:1035] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f0941f30504963adf42edd0e42bb66" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 43823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:00.219794 11804 mvcc.cc:205] Tried to move back new op lower bound from 7334204621682802688 to 7334204621548396544. Current Snapshot: MvccSnapshot[applied={T|T < 7334204621682802688}]
I20260928 06:35:00.221454 11803 mvcc.cc:205] Tried to move back new op lower bound from 7334204621682802688 to 7334204621548396544. Current Snapshot: MvccSnapshot[applied={T|T < 7334204621682802688 or (T in {7334204621682802688})}]
I20260928 06:35:00.221781 11312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:00.221884 11312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:00.222695 11312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d185fef5f2ea427dbf08b5b29b571f65: Sending DeleteTablet for 3 replicas of tablet 05de6a954bca42fea5e9b6b4eacb42d0
I20260928 06:35:00.223037 11706 tablet_service.cc:1514] Processing DeleteTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:47370
I20260928 06:35:00.223042 11438 tablet_service.cc:1514] Processing DeleteTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:56406
I20260928 06:35:00.223366 11811 tablet_replica.cc:330] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66: stopping tablet replica
I20260928 06:35:00.223433 11572 tablet_service.cc:1514] Processing DeleteTablet for tablet 05de6a954bca42fea5e9b6b4eacb42d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='kudu', principal='kudu/127.24.11.190@KRBTEST.COM'} at 127.0.0.1:34828
I20260928 06:35:00.223485 11811 raft_consensus.cc:2229] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:00.223593 11811 raft_consensus.cc:2258] T 05de6a954bca42fea5e9b6b4eacb42d0 P b0f0941f30504963adf42edd0e42bb66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:00.223695 11813 tablet_replica.cc:330] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: stopping tablet replica
I20260928 06:35:00.223771 11812 tablet_replica.cc:330] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: stopping tablet replica
I20260928 06:35:00.223834 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11375
I20260928 06:35:00.223875 11812 raft_consensus.cc:2229] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:00.223874 11813 raft_consensus.cc:2229] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:35:00.223966 11812 raft_consensus.cc:2258] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:00.224105 11813 raft_consensus.cc:2258] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:00.224604 11813 ts_tablet_manager.cc:1905] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:00.226824 11813 ts_tablet_manager.cc:1918] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:00.226873 11812 ts_tablet_manager.cc:1905] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:00.227002 11813 log.cc:1196] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/05de6a954bca42fea5e9b6b4eacb42d0
I20260928 06:35:00.227361 11813 ts_tablet_manager.cc:1939] T 05de6a954bca42fea5e9b6b4eacb42d0 P 82c824706cb340509b84d079c8ae16d6: Deleting consensus metadata
I20260928 06:35:00.227957 11297 catalog_manager.cc:4841] TS 82c824706cb340509b84d079c8ae16d6 (127.24.11.130:40481): tablet 05de6a954bca42fea5e9b6b4eacb42d0 (table test-table [id=c0daec46caa345399a6b3948ce5d79e7]) successfully deleted
I20260928 06:35:00.228530 11812 ts_tablet_manager.cc:1918] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:00.228609 11812 log.cc:1196] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/05de6a954bca42fea5e9b6b4eacb42d0
I20260928 06:35:00.228900 11812 ts_tablet_manager.cc:1939] T 05de6a954bca42fea5e9b6b4eacb42d0 P 20395c3a27ce4c78bcfa06dca557b5a2: Deleting consensus metadata
I20260928 06:35:00.229350 11300 catalog_manager.cc:4841] TS 20395c3a27ce4c78bcfa06dca557b5a2 (127.24.11.131:45559): tablet 05de6a954bca42fea5e9b6b4eacb42d0 (table test-table [id=c0daec46caa345399a6b3948ce5d79e7]) successfully deleted
W20260928 06:35:00.231009 11298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:43823 (error 108)
I20260928 06:35:00.231235 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11509
I20260928 06:35:00.231953 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.231506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:43823 (local address 127.0.0.1:56422)
0928 06:35:00.231871 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.231912 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:43823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340}
W20260928 06:35:00.232530 11298 catalog_manager.cc:4584] TS b0f0941f30504963adf42edd0e42bb66 (127.24.11.129:43823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05de6a954bca42fea5e9b6b4eacb42d0: Network error: Client connection negotiation failed: client connection to 127.24.11.129:43823: connect: Connection refused (error 111)
I20260928 06:35:00.236732 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11643
I20260928 06:35:00.241585 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11281
2026-09-28T06:35:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2055 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:00Z Disabled control of system clock
I20260928 06:35:00.269619 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43423
--webserver_interface=127.24.11.190
--webserver_port=0
--builtin_ntp_servers=127.24.11.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:35:00.352207 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:00.352463 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:00.352485 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:00.353873 11818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:00.353922 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:00.353935 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:00.353947 11818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:00.353960 11818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:00.355403 11818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:00.355690 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:00.356009 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:00.361095 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.361102 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.361102 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:00.361310 11818 server_base.cc:1034] running on GCE node
I20260928 06:35:00.361666 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:00.362001 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:00.363184 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790577300363157 us; error 41 us; skew 500 ppm
I20260928 06:35:00.364503 11818 webserver.cc:460] Webserver started at http://127.24.11.190:45959/ using document root <none> and password file <none>
I20260928 06:35:00.364778 11818 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:00.364833 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:00.364987 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:00.366207 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "c9429fc6bef449eea3f1be93d959f9c9"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "f86738c1f3e12a9164d9a77ff6edfacd"
server_key_iv: "e27c16cf595b6a43e86e8ab03487267c"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.366604 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "c9429fc6bef449eea3f1be93d959f9c9"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "f86738c1f3e12a9164d9a77ff6edfacd"
server_key_iv: "e27c16cf595b6a43e86e8ab03487267c"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.368007 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:00.368976 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.369230 11818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:00.369333 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "c9429fc6bef449eea3f1be93d959f9c9"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "f86738c1f3e12a9164d9a77ff6edfacd"
server_key_iv: "e27c16cf595b6a43e86e8ab03487267c"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.369416 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:00.395764 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:00.396147 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:00.396319 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:00.400558 11818 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43423
I20260928 06:35:00.400624 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43423 every 8 connection(s)
I20260928 06:35:00.400966 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:00.402097 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.404481 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11818
I20260928 06:35:00.404577 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:00.404701 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: Bootstrap starting.
I20260928 06:35:00.404796 24622 external_mini_cluster.cc:1428] Setting key d24d12ebd9cb00bb4ef38d55dcc7d0e7
I20260928 06:35:00.405514 11885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.405893 11885 log.cc:824] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.406704 11885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: No bootstrap required, opened a new log
I20260928 06:35:00.409008 11885 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } }
I20260928 06:35:00.409179 11885 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.409214 11885 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9429fc6bef449eea3f1be93d959f9c9, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.409359 11885 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } }
I20260928 06:35:00.409436 11885 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:00.409473 11885 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:00.409519 11885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.410336 11885 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } }
I20260928 06:35:00.410473 11885 leader_election.cc:302] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9429fc6bef449eea3f1be93d959f9c9; no voters:
I20260928 06:35:00.410668 11885 leader_election.cc:288] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:00.410741 11890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:00.410934 11890 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [term 1 LEADER]: Becoming Leader. State: Replica: c9429fc6bef449eea3f1be93d959f9c9, State: Running, Role: LEADER
I20260928 06:35:00.410972 11885 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:00.411020 11890 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } }
I20260928 06:35:00.411357 11891 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9429fc6bef449eea3f1be93d959f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } } }
I20260928 06:35:00.411464 11891 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:00.411635 11892 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9429fc6bef449eea3f1be93d959f9c9. Latest consensus state: current_term: 1 leader_uuid: "c9429fc6bef449eea3f1be93d959f9c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9429fc6bef449eea3f1be93d959f9c9" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43423 } } }
I20260928 06:35:00.411705 11892 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:00.411396 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.405390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37136 (local address 127.24.11.190:43423)
0928 06:35:00.405655 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.405666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.405707 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:35:00.406277 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.406285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.406390 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.406542 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.406722 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.408108 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.408919 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.409583 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.409710 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.410431 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.410436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.410436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:00.410659 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.410706 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:00.410723 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.410794 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.411080 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.411173 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":83,"threads_started":1}
I20260928 06:35:00.411676 11896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:00.412350 11896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:00.414595 11896 catalog_manager.cc:1347] Generated new cluster ID: b0d04201f63a47c8ac423091788047ba
I20260928 06:35:00.414687 11896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:00.433885 11896 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:00.434767 11896 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:00.437132 11896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: Generated new TSK 0
I20260928 06:35:00.437350 11896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:00.438763 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.129:0
--local_ip_for_outbound_sockets=127.24.11.129
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--builtin_ntp_servers=127.24.11.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:35:00.523190 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:00.523456 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:00.523495 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:00.524765 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:00.524834 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:00.524878 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.129
I20260928 06:35:00.526289 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.11.129
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.129
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:00.526561 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:00.526875 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:00.532960 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.532960 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.532986 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:00.533226 11909 server_base.cc:1034] running on GCE node
I20260928 06:35:00.533463 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:00.533793 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:00.534987 11909 hybrid_clock.cc:648] HybridClock initialized: now 1790577300534965 us; error 37 us; skew 500 ppm
I20260928 06:35:00.536482 11909 webserver.cc:460] Webserver started at http://127.24.11.129:36053/ using document root <none> and password file <none>
I20260928 06:35:00.536779 11909 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:00.536845 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:00.536988 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:00.538576 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data/instance:
uuid: "949e9b5509704d119c60eae80e59688b"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "e65c7047c043df9f6352219941f74c04"
server_key_iv: "c05d05be25fa92a5faabf524b254ee14"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.539009 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance:
uuid: "949e9b5509704d119c60eae80e59688b"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "e65c7047c043df9f6352219941f74c04"
server_key_iv: "c05d05be25fa92a5faabf524b254ee14"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.540979 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:35:00.541795 11923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.541965 11909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:00.542073 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal
uuid: "949e9b5509704d119c60eae80e59688b"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "e65c7047c043df9f6352219941f74c04"
server_key_iv: "c05d05be25fa92a5faabf524b254ee14"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.542151 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:00.552641 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:00.552994 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:00.553158 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:00.553407 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:00.553817 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:00.553874 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.553921 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:00.553946 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.559849 11909 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.129:36363
I20260928 06:35:00.559927 12036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.129:36363 every 8 connection(s)
I20260928 06:35:00.560276 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/data/info.pb
I20260928 06:35:00.564555 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 11909
I20260928 06:35:00.564688 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-0/wal/instance
I20260928 06:35:00.564935 24622 external_mini_cluster.cc:1428] Setting key cc765a6dea69f5b549780bb36bdd662e
I20260928 06:35:00.565980 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.561051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:41869 (local address 127.24.11.190:43423)
0928 06:35:00.561120 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.561122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.561352 (+ 230us) server_negotiation.cc:407] Connection header received
0928 06:35:00.561997 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.561999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.562013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.562062 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.562754 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.563271 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.564285 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.564492 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.564571 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.565748 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.565749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.565750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:00.565755 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.565770 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:00.565774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.565818 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.565900 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.565922 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:35:00.565976 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.130:0
--local_ip_for_outbound_sockets=127.24.11.130
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--builtin_ntp_servers=127.24.11.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:35:00.566319 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.561052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43423 (local address 127.24.11.129:41869)
0928 06:35:00.561228 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.561263 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.561879 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.562140 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.562143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.562152 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:00.562653 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.562657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.563374 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.563376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.564178 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.564180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.564581 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.564582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.564765 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.565585 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:00.565587 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:00.565618 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:00.565664 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:00.565822 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:00.565827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:00.565830 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.565900 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.566020 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260928 06:35:00.566967 12037 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43423
I20260928 06:35:00.567067 12037 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:00.567312 12037 heartbeater.cc:511] Master 127.24.11.190:43423 requested a full tablet report, sending...
I20260928 06:35:00.567972 11849 ts_manager.cc:194] Registered new tserver with Master: 949e9b5509704d119c60eae80e59688b (127.24.11.129:36363)
I20260928 06:35:00.568729 11849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.129:41869
W20260928 06:35:00.649645 12040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:00.649865 12040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:00.649889 12040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:00.651048 12040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:00.651088 12040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:00.651113 12040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.130
I20260928 06:35:00.652416 12040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.11.130
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.130
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:00.652616 12040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:00.652882 12040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:00.658505 12046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.658553 12045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.658479 12048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:00.658794 12040 server_base.cc:1034] running on GCE node
I20260928 06:35:00.659037 12040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:00.659303 12040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:00.660472 12040 hybrid_clock.cc:648] HybridClock initialized: now 1790577300660462 us; error 49 us; skew 500 ppm
I20260928 06:35:00.661990 12040 webserver.cc:460] Webserver started at http://127.24.11.130:32951/ using document root <none> and password file <none>
I20260928 06:35:00.662245 12040 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:00.662317 12040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:00.662472 12040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:00.663620 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data/instance:
uuid: "2b08deef9b3f476a9e122f7ddc349a87"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "2080cea949fa62235af7495c58ca27ad"
server_key_iv: "6819c18afdfc1028a14fae4ac69354b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.664003 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance:
uuid: "2b08deef9b3f476a9e122f7ddc349a87"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "2080cea949fa62235af7495c58ca27ad"
server_key_iv: "6819c18afdfc1028a14fae4ac69354b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.665405 12040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:00.666332 12054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.666519 12040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:00.666627 12040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal
uuid: "2b08deef9b3f476a9e122f7ddc349a87"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "2080cea949fa62235af7495c58ca27ad"
server_key_iv: "6819c18afdfc1028a14fae4ac69354b1"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.666849 12040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:00.679988 12040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:00.680334 12040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:00.680495 12040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:00.680735 12040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:00.681110 12040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:00.681167 12040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.681212 12040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:00.681237 12040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.687386 12040 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.130:38555
I20260928 06:35:00.687429 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.130:38555 every 8 connection(s)
I20260928 06:35:00.687774 12040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/data/info.pb
I20260928 06:35:00.692173 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12040
I20260928 06:35:00.692287 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal/instance
I20260928 06:35:00.692517 24622 external_mini_cluster.cc:1428] Setting key 0aaae48363d0480970dd637672e00d87
I20260928 06:35:00.693176 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.688518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.130:60011 (local address 127.24.11.190:43423)
0928 06:35:00.688592 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.688594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.688865 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:35:00.689324 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.689325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.689336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.689396 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.690106 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.690628 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.691692 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.691888 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.692003 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.692876 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.692877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.692877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:00.692881 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.692900 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:00.692905 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.692961 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.693072 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.693099 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:35:00.693430 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.688509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43423 (local address 127.24.11.130:60011)
0928 06:35:00.688716 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.688758 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.689229 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.689448 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.689451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.689459 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:00.690023 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.690028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.690806 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.690807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.691593 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.691595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.691975 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.691976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.692086 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.692704 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:00.692705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:00.692757 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:00.692789 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:00.692976 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:00.692984 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:00.692987 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.693075 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.693182 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260928 06:35:00.693714 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.11.131:0
--local_ip_for_outbound_sockets=127.24.11.131
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--builtin_ntp_servers=127.24.11.148:38079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:35:00.694031 12168 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43423
I20260928 06:35:00.694137 12168 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:00.694324 12168 heartbeater.cc:511] Master 127.24.11.190:43423 requested a full tablet report, sending...
I20260928 06:35:00.694718 11849 ts_manager.cc:194] Registered new tserver with Master: 2b08deef9b3f476a9e122f7ddc349a87 (127.24.11.130:38555)
I20260928 06:35:00.695226 11849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.130:60011
W20260928 06:35:00.776221 12171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:00.776435 12171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:00.776458 12171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:00.777688 12171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:00.777730 12171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:00.777763 12171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.11.131
I20260928 06:35:00.779213 12171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:38079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.11.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.11.131
--webserver_port=0
--tserver_master_addrs=127.24.11.190:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.11.131
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:00.779495 12171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:00.779839 12171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:00.785594 12171 server_base.cc:1034] running on GCE node
W20260928 06:35:00.785563 12177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.785557 12179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:00.785579 12176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:00.786077 12171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:00.786401 12171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:00.787596 12171 hybrid_clock.cc:648] HybridClock initialized: now 1790577300787549 us; error 62 us; skew 500 ppm
I20260928 06:35:00.789050 12171 webserver.cc:460] Webserver started at http://127.24.11.131:33705/ using document root <none> and password file <none>
I20260928 06:35:00.789322 12171 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:00.789397 12171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:00.789603 12171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:00.790948 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data/instance:
uuid: "db2cd79dcac94bd2957d93f4719eeb5c"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "78d9a82f49411dc5370bb870a9466cb0"
server_key_iv: "711aecfa271236086e173822a8b8a219"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.791383 12171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance:
uuid: "db2cd79dcac94bd2957d93f4719eeb5c"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "78d9a82f49411dc5370bb870a9466cb0"
server_key_iv: "711aecfa271236086e173822a8b8a219"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.792865 12171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:00.793920 12185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.794188 12171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:00.794302 12171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal
uuid: "db2cd79dcac94bd2957d93f4719eeb5c"
format_stamp: "Formatted at 2026-09-28 06:35:00 on dist-test-slave-38pg"
server_key: "78d9a82f49411dc5370bb870a9466cb0"
server_key_iv: "711aecfa271236086e173822a8b8a219"
server_key_version: "encryptionkey@0"
I20260928 06:35:00.794399 12171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:00.812595 12171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:00.812932 12171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:00.813090 12171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:00.813341 12171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:35:00.813740 12171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:35:00.813830 12171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.813874 12171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:35:00.813905 12171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:00.819830 12171 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.131:41905
I20260928 06:35:00.819864 12298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.131:41905 every 8 connection(s)
I20260928 06:35:00.820209 12171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/data/info.pb
I20260928 06:35:00.825297 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.820975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.131:50415 (local address 127.24.11.190:43423)
0928 06:35:00.821043 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.821049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.821261 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:35:00.821817 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.821817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.821827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.821889 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.822534 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.823025 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.824002 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.824181 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.824232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.825005 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.825006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.825007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:00.825011 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.825025 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:00.825034 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.825078 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.825215 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.825238 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:00.825629 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.820947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:43423 (local address 127.24.11.131:50415)
0928 06:35:00.821132 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.821174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.821724 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.821944 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.821947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.821955 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:00.822461 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.822464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.823110 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.823111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.823919 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.823922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.824252 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.824252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.824372 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.824868 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:35:00.824869 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:00.824896 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:35:00.824919 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:00.825124 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:00.825130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:00.825134 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.825208 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.825306 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260928 06:35:00.826166 12299 heartbeater.cc:347] Connected to a master server at 127.24.11.190:43423
I20260928 06:35:00.826273 12299 heartbeater.cc:464] Registering TS with master...
I20260928 06:35:00.826471 12299 heartbeater.cc:511] Master 127.24.11.190:43423 requested a full tablet report, sending...
I20260928 06:35:00.826885 11849 ts_manager.cc:194] Registered new tserver with Master: db2cd79dcac94bd2957d93f4719eeb5c (127.24.11.131:41905)
I20260928 06:35:00.827298 11849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.11.131:50415
I20260928 06:35:00.829003 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12171
I20260928 06:35:00.829088 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal/instance
I20260928 06:35:00.829239 24622 external_mini_cluster.cc:1428] Setting key 52f38205636b37ef1d21925a836c469a
I20260928 06:35:00.830137 24622 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:35:00.834365 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.831425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37140 (local address 127.24.11.190:43423)
0928 06:35:00.831506 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.831508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.831608 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:35:00.831679 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.831680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.831690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.831740 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:00.831904 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.832380 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.833141 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.833333 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.833408 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.834124 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:00.834126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:00.834126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:00.834131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:00.834146 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:00.834150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:00.834205 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.834258 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.834281 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 06:35:00.836035 11849 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:35:00.837741 11849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:35:00.844430 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:38555 (local address 127.0.0.1:59882)
0928 06:35:00.840218 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.840262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.840339 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.840662 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.840664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.840667 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.840738 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.840741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.841934 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.841935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.843341 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.843344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.844114 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.844115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.844195 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.844211 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.844278 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.844342 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":74,"thread_start_us":79,"threads_started":1}
I20260928 06:35:00.844539 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.24.11.131:41905)
0928 06:35:00.840788 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.840792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.840802 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:00.840839 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.840841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.840856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.841753 (+ 897us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.841818 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.842547 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.843646 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.844180 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.844296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.844384 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.844410 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.844449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":89,"threads_started":1}
I20260928 06:35:00.844549 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41905 (local address 127.0.0.1:41606)
0928 06:35:00.840295 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.840322 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.840375 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.841640 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.841641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.841643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.841682 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.841683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.842636 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.842637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.843565 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.843567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.844274 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.844275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.844343 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.844347 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.844393 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.844430 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
I20260928 06:35:00.844933 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.24.11.130:38555)
0928 06:35:00.840500 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.840505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.840514 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:00.840557 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.840559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.840571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.840658 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.840951 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.841849 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.843433 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.844023 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.844079 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.844142 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.844792 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.844838 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":84,"threads_started":1}
I20260928 06:35:00.845408 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36363 (local address 127.0.0.1:33664)
0928 06:35:00.840373 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.840383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.840425 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.840817 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.840818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.840820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.840866 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.840867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.841973 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.841974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.843341 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.843344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.845194 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.845195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.845253 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.845258 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.845303 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.845343 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":43,"thread_start_us":80,"threads_started":1}
I20260928 06:35:00.845855 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.840461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.24.11.129:36363)
0928 06:35:00.840649 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.840654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.840664 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:00.840708 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.840709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.840721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.840813 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.840975 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.841909 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.844174 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.844990 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.845118 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.845634 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.845659 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.845701 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260928 06:35:00.845743 12233 tablet_service.cc:1467] Processing CreateTablet for tablet c8c22f922451466e9dac368373dcc1dc (DEFAULT_TABLE table=test-table [id=fae2047a80514233bd98225b61bd0a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.845741 12102 tablet_service.cc:1467] Processing CreateTablet for tablet c8c22f922451466e9dac368373dcc1dc (DEFAULT_TABLE table=test-table [id=fae2047a80514233bd98225b61bd0a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.846122 12233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c22f922451466e9dac368373dcc1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.846123 12102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c22f922451466e9dac368373dcc1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.848852 12313 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Bootstrap starting.
I20260928 06:35:00.849134 12314 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Bootstrap starting.
I20260928 06:35:00.849246 11971 tablet_service.cc:1467] Processing CreateTablet for tablet c8c22f922451466e9dac368373dcc1dc (DEFAULT_TABLE table=test-table [id=fae2047a80514233bd98225b61bd0a91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:35:00.849648 11971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c22f922451466e9dac368373dcc1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:00.849664 12313 tablet_bootstrap.cc:654] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.849838 12314 tablet_bootstrap.cc:654] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.850006 12313 log.cc:824] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.850145 12314 log.cc:824] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.850821 12314 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: No bootstrap required, opened a new log
I20260928 06:35:00.850842 12313 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: No bootstrap required, opened a new log
I20260928 06:35:00.850907 12313 ts_tablet_manager.cc:1397] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:00.850908 12314 ts_tablet_manager.cc:1397] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:00.852876 12314 raft_consensus.cc:348] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.853058 12314 raft_consensus.cc:374] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.853106 12314 raft_consensus.cc:729] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b08deef9b3f476a9e122f7ddc349a87, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.853264 12317 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: Bootstrap starting.
I20260928 06:35:00.853240 12314 consensus_queue.cc:260] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.853569 12314 ts_tablet_manager.cc:1428] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:35:00.853569 12313 raft_consensus.cc:348] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.853720 12168 heartbeater.cc:503] Master 127.24.11.190:43423 was elected leader, sending a full tablet report...
I20260928 06:35:00.853724 12313 raft_consensus.cc:374] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.853760 12313 raft_consensus.cc:729] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2cd79dcac94bd2957d93f4719eeb5c, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.853870 12313 consensus_queue.cc:260] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.853976 12317 tablet_bootstrap.cc:654] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:00.854108 12313 ts_tablet_manager.cc:1428] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:35:00.854169 12299 heartbeater.cc:503] Master 127.24.11.190:43423 was elected leader, sending a full tablet report...
I20260928 06:35:00.854297 12317 log.cc:824] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:00.855160 12317 tablet_bootstrap.cc:492] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: No bootstrap required, opened a new log
I20260928 06:35:00.855247 12317 ts_tablet_manager.cc:1397] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:35:00.857053 12317 raft_consensus.cc:348] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.857200 12317 raft_consensus.cc:374] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:00.857245 12317 raft_consensus.cc:729] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949e9b5509704d119c60eae80e59688b, State: Initialized, Role: FOLLOWER
I20260928 06:35:00.857359 12317 consensus_queue.cc:260] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.857648 12317 ts_tablet_manager.cc:1428] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 06:35:00.857692 12037 heartbeater.cc:503] Master 127.24.11.190:43423 was elected leader, sending a full tablet report...
I20260928 06:35:00.937037 12321 raft_consensus.cc:482] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:35:00.937179 12321 raft_consensus.cc:504] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.937502 12321 leader_election.cc:288] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b08deef9b3f476a9e122f7ddc349a87 (127.24.11.130:38555), db2cd79dcac94bd2957d93f4719eeb5c (127.24.11.131:41905)
W20260928 06:35:00.938277 12169 tablet.cc:2369] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:35:00.941344 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.937842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:48615 (local address 127.24.11.131:41905)
0928 06:35:00.937888 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.937890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.937900 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:00.937969 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.937970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.937985 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.938048 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.938213 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.938969 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.940406 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.940962 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.941048 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.941057 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.941232 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.941272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:00.941342 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.937843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.11.129:51999 (local address 127.24.11.130:38555)
0928 06:35:00.937892 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.937894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.938012 (+ 118us) server_negotiation.cc:407] Connection header received
0928 06:35:00.938093 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.938094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.938102 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.938168 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:35:00.938376 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.939102 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.940406 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.940996 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.941061 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.941068 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.941233 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.941272 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:00.941366 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.937727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.131:41905 (local address 127.24.11.129:48615)
0928 06:35:00.937813 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.937828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.937892 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.938050 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.938052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.938053 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.938129 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.938131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.939065 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.939066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.940297 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.940309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.941068 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.941069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.941150 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.941156 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.941215 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.941258 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":8}
I20260928 06:35:00.941373 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.937729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.130:38555 (local address 127.24.11.129:51999)
0928 06:35:00.937931 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.937941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:00.938026 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:00.938214 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:00.938215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:00.938217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:35:00.938292 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.938294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.939172 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.939173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.940299 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:00.940300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.941068 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.941069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:00.941150 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.941156 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:35:00.941215 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:35:00.941257 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":69,"thread_start_us":101,"threads_started":1}
I20260928 06:35:00.941819 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c22f922451466e9dac368373dcc1dc" candidate_uuid: "949e9b5509704d119c60eae80e59688b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" is_pre_election: true
I20260928 06:35:00.941998 12253 raft_consensus.cc:2454] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949e9b5509704d119c60eae80e59688b in term 0.
I20260928 06:35:00.941998 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c22f922451466e9dac368373dcc1dc" candidate_uuid: "949e9b5509704d119c60eae80e59688b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b08deef9b3f476a9e122f7ddc349a87" is_pre_election: true
I20260928 06:35:00.942167 12122 raft_consensus.cc:2454] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 949e9b5509704d119c60eae80e59688b in term 0.
I20260928 06:35:00.942276 11925 leader_election.cc:302] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949e9b5509704d119c60eae80e59688b, db2cd79dcac94bd2957d93f4719eeb5c; no voters:
I20260928 06:35:00.942417 12321 raft_consensus.cc:2781] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:35:00.942489 12321 raft_consensus.cc:482] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:35:00.942507 12321 raft_consensus.cc:3037] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.943238 12321 raft_consensus.cc:504] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.943383 12321 leader_election.cc:288] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [CANDIDATE]: Term 1 election: Requested vote from peers 2b08deef9b3f476a9e122f7ddc349a87 (127.24.11.130:38555), db2cd79dcac94bd2957d93f4719eeb5c (127.24.11.131:41905)
I20260928 06:35:00.943516 12122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c22f922451466e9dac368373dcc1dc" candidate_uuid: "949e9b5509704d119c60eae80e59688b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b08deef9b3f476a9e122f7ddc349a87"
I20260928 06:35:00.943559 12253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c22f922451466e9dac368373dcc1dc" candidate_uuid: "949e9b5509704d119c60eae80e59688b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2cd79dcac94bd2957d93f4719eeb5c"
I20260928 06:35:00.943603 12122 raft_consensus.cc:3037] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.943641 12253 raft_consensus.cc:3037] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:00.944538 12253 raft_consensus.cc:2454] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949e9b5509704d119c60eae80e59688b in term 1.
I20260928 06:35:00.944548 12122 raft_consensus.cc:2454] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 949e9b5509704d119c60eae80e59688b in term 1.
I20260928 06:35:00.944697 11925 leader_election.cc:302] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 949e9b5509704d119c60eae80e59688b, db2cd79dcac94bd2957d93f4719eeb5c; no voters:
I20260928 06:35:00.944797 12321 raft_consensus.cc:2781] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:00.944986 12321 raft_consensus.cc:686] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [term 1 LEADER]: Becoming Leader. State: Replica: 949e9b5509704d119c60eae80e59688b, State: Running, Role: LEADER
I20260928 06:35:00.945094 12321 consensus_queue.cc:237] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } }
I20260928 06:35:00.946060 11848 catalog_manager.cc:5495] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b reported cstate change: term changed from 0 to 1, leader changed from <none> to 949e9b5509704d119c60eae80e59688b (127.24.11.129). New cstate: current_term: 1 leader_uuid: "949e9b5509704d119c60eae80e59688b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "949e9b5509704d119c60eae80e59688b" member_type: VOTER last_known_addr { host: "127.24.11.129" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260928 06:35:00.960245 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.957842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.24.11.129:36363)
0928 06:35:00.957907 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:00.957909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:00.957918 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:00.957957 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:00.957958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:00.957987 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:00.958066 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:00.958216 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.958698 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.959439 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:00.959641 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:00.959718 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:00.959885 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:00.959991 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:00.960089 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:00.960113 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 06:35:00.960133 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:00.962059 12122 raft_consensus.cc:1261] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 1 FOLLOWER]: Refusing update from remote peer 949e9b5509704d119c60eae80e59688b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:00.962069 12253 raft_consensus.cc:1261] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 1 FOLLOWER]: Refusing update from remote peer 949e9b5509704d119c60eae80e59688b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:00.962450 12321 consensus_queue.cc:1035] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b08deef9b3f476a9e122f7ddc349a87" member_type: VOTER last_known_addr { host: "127.24.11.130" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:00.962606 12323 consensus_queue.cc:1035] T c8c22f922451466e9dac368373dcc1dc P 949e9b5509704d119c60eae80e59688b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2cd79dcac94bd2957d93f4719eeb5c" member_type: VOTER last_known_addr { host: "127.24.11.131" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:00.965005 12326 mvcc.cc:205] Tried to move back new op lower bound from 7334204624737726464 to 7334204624671764480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204624737726464}]
I20260928 06:35:00.965226 12328 mvcc.cc:205] Tried to move back new op lower bound from 7334204624737726464 to 7334204624671764480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204624737726464}]
I20260928 06:35:00.965320 12327 mvcc.cc:205] Tried to move back new op lower bound from 7334204624737726464 to 7334204624671764480. Current Snapshot: MvccSnapshot[applied={T|T < 7334204624737726464}]
I20260928 06:35:00.967411 11848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:00.967511 11848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:35:00.968302 11848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9429fc6bef449eea3f1be93d959f9c9: Sending DeleteTablet for 3 replicas of tablet c8c22f922451466e9dac368373dcc1dc
I20260928 06:35:00.968654 12233 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c22f922451466e9dac368373dcc1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='slave'} at 127.0.0.1:41606
I20260928 06:35:00.968683 12102 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c22f922451466e9dac368373dcc1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='slave'} at 127.0.0.1:59882
I20260928 06:35:00.968871 11971 tablet_service.cc:1514] Processing DeleteTablet for tablet c8c22f922451466e9dac368373dcc1dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:35:00 UTC) from {username='slave'} at 127.0.0.1:33664
I20260928 06:35:00.968973 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11909
I20260928 06:35:00.969092 12337 tablet_replica.cc:330] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: stopping tablet replica
I20260928 06:35:00.969192 12337 raft_consensus.cc:2229] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:00.969303 12337 raft_consensus.cc:2258] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:00.969853 12337 ts_tablet_manager.cc:1905] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:00.971843 12336 tablet_replica.cc:330] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: stopping tablet replica
I20260928 06:35:00.971987 12336 raft_consensus.cc:2229] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:35:00.972061 12337 ts_tablet_manager.cc:1918] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:00.972105 12336 raft_consensus.cc:2258] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:35:00.972136 12337 log.cc:1196] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-1/wal/wals/c8c22f922451466e9dac368373dcc1dc
I20260928 06:35:00.972435 12337 ts_tablet_manager.cc:1939] T c8c22f922451466e9dac368373dcc1dc P 2b08deef9b3f476a9e122f7ddc349a87: Deleting consensus metadata
I20260928 06:35:00.972527 12336 ts_tablet_manager.cc:1905] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:35:00.972863 11833 catalog_manager.cc:4841] TS 2b08deef9b3f476a9e122f7ddc349a87 (127.24.11.130:38555): tablet c8c22f922451466e9dac368373dcc1dc (table test-table [id=fae2047a80514233bd98225b61bd0a91]) successfully deleted
I20260928 06:35:00.974532 12336 ts_tablet_manager.cc:1918] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:35:00.974613 12336 log.cc:1196] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Deleting WAL directory at /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577055356400-24622-0/minicluster-data/ts-2/wal/wals/c8c22f922451466e9dac368373dcc1dc
I20260928 06:35:00.974879 12336 ts_tablet_manager.cc:1939] T c8c22f922451466e9dac368373dcc1dc P db2cd79dcac94bd2957d93f4719eeb5c: Deleting consensus metadata
I20260928 06:35:00.975427 11834 catalog_manager.cc:4841] TS db2cd79dcac94bd2957d93f4719eeb5c (127.24.11.131:41905): tablet c8c22f922451466e9dac368373dcc1dc (table test-table [id=fae2047a80514233bd98225b61bd0a91]) successfully deleted
W20260928 06:35:00.976023 11833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.11.129:36363 (error 108)
I20260928 06:35:00.976234 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12040
I20260928 06:35:00.976524 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:00.976399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.129:36363 (local address 127.0.0.1:33684)
0928 06:35:00.976448 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:00.976485 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:35:00.976630 11833 catalog_manager.cc:4584] TS 949e9b5509704d119c60eae80e59688b (127.24.11.129:36363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8c22f922451466e9dac368373dcc1dc: Network error: Client connection negotiation failed: client connection to 127.24.11.129:36363: connect: Connection refused (error 111)
I20260928 06:35:00.981473 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12171
I20260928 06:35:00.985914 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 11818
2026-09-28T06:35:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (743 ms)
[----------] 6 tests from EncryptionPolicyTest (8687 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:01 dist-test-slave-38pg krb5kdc[12342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:01 dist-test-slave-38pg krb5kdc[12342](info): set up 2 sockets
Sep 28 06:35:01 dist-test-slave-38pg krb5kdc[12342](info): commencing operation
krb5kdc: starting...
W20260928 06:35:02.018860 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:02.142876 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:02.223973 12355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:02.224236 12355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:02.224285 12355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:02.225467 12355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:02.225541 12355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:02.225611 12355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:02.225644 12355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:02.225672 12355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:02.227177 12355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:02.227463 12355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:02.227692 12355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:02.232762 12361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:02.232843 12360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:02.232991 12363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:02.233007 12355 server_base.cc:1034] running on GCE node
I20260928 06:35:02.233323 12355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:02.233703 12355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:02.234905 12355 hybrid_clock.cc:648] HybridClock initialized: now 1790577302234886 us; error 30 us; skew 500 ppm
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:02.237777 12355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:02.238313 12355 webserver.cc:460] Webserver started at http://127.0.0.1:43001/ using document root <none> and password file <none>
I20260928 06:35:02.238556 12355 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:02.238628 12355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:02.238790 12355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:02.239917 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "4f75e8e65bf249959f673801b1e2f3ad"
format_stamp: "Formatted at 2026-09-28 06:35:02 on dist-test-slave-38pg"
server_key: "917c6102a30302102cca2104160441fc"
server_key_iv: "9c40f2db3c6d20fa55521caf891f2e03"
server_key_version: "encryptionkey@0"
I20260928 06:35:02.240321 12355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "4f75e8e65bf249959f673801b1e2f3ad"
format_stamp: "Formatted at 2026-09-28 06:35:02 on dist-test-slave-38pg"
server_key: "917c6102a30302102cca2104160441fc"
server_key_iv: "9c40f2db3c6d20fa55521caf891f2e03"
server_key_version: "encryptionkey@0"
I20260928 06:35:02.241871 12355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 06:35:02.242633 12370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:02.242812 12355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:02.242920 12355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "4f75e8e65bf249959f673801b1e2f3ad"
format_stamp: "Formatted at 2026-09-28 06:35:02 on dist-test-slave-38pg"
server_key: "917c6102a30302102cca2104160441fc"
server_key_iv: "9c40f2db3c6d20fa55521caf891f2e03"
server_key_version: "encryptionkey@0"
I20260928 06:35:02.242995 12355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:02.257797 12355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:02.259924 12355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:02.260104 12355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:02.264475 12355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38225
I20260928 06:35:02.264513 12422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38225 every 8 connection(s)
I20260928 06:35:02.264878 12355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:02.265759 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:02.267947 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: Bootstrap starting.
I20260928 06:35:02.267982 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12355
I20260928 06:35:02.268121 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:02.268329 24622 external_mini_cluster.cc:1428] Setting key bb564b288929283a06e00b2e3c2e6bd6
I20260928 06:35:02.268657 12423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:02.269013 12423 log.cc:824] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:02.269871 12423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: No bootstrap required, opened a new log
I20260928 06:35:02.272204 12423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260928 06:35:02.272388 12423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:02.272418 12423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f75e8e65bf249959f673801b1e2f3ad, State: Initialized, Role: FOLLOWER
I20260928 06:35:02.272570 12423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260928 06:35:02.272652 12423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:02.272675 12423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:02.272749 12423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:02.273449 12423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260928 06:35:02.273581 12423 leader_election.cc:302] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f75e8e65bf249959f673801b1e2f3ad; no voters:
I20260928 06:35:02.273742 12423 leader_election.cc:288] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:02.273805 12428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:02.273994 12428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [term 1 LEADER]: Becoming Leader. State: Replica: 4f75e8e65bf249959f673801b1e2f3ad, State: Running, Role: LEADER
I20260928 06:35:02.274001 12423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:02.274780 12428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } }
I20260928 06:35:02.275437 12429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f75e8e65bf249959f673801b1e2f3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } } }
I20260928 06:35:02.275564 12429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:02.275885 12430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f75e8e65bf249959f673801b1e2f3ad. Latest consensus state: current_term: 1 leader_uuid: "4f75e8e65bf249959f673801b1e2f3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f75e8e65bf249959f673801b1e2f3ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38225 } } }
W20260928 06:35:02.276065 12442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 06:35:02.276100 12430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:02.276113 12442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:02.276396 12443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:35:02.276607 12443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:02.278213 12443 catalog_manager.cc:1347] Generated new cluster ID: 0b0dc2106fff412c93597fbc177a8939
I20260928 06:35:02.278311 12443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:02.284471 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:02.268927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51743 (local address 127.0.0.1:38225)
0928 06:35:02.269143 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:02.269152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:02.269187 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:02.269800 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:02.269809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:02.269870 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:02.270025 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:02.270682 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:02.272186 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:02.273158 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:02.273855 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:02.274338 (+ 483us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:02.277095 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:02.277099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:02.277100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:02.277113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:02.282548 (+ 5435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:02.283107 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:02.283111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:02.283111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:02.283239 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:02.283496 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:02.283497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:02.283497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:02.283790 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:02.283933 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:02.284145 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:35:02.284246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260928 06:35:02.285632 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:02.294555 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:02.287090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36629 (local address 127.0.0.1:38225)
0928 06:35:02.287128 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:02.287130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:02.287207 (+ 77us) server_negotiation.cc:407] Connection header received
0928 06:35:02.287251 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:02.287252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:02.287262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:02.287320 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:02.287895 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:02.288679 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:02.289705 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:02.290053 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:02.290121 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:02.291634 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:02.291635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:02.291636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:02.291643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:02.293239 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:02.293728 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:02.293730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:02.293730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:02.293788 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:02.294021 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:02.294021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:02.294022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:02.294142 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:02.294234 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:02.294398 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:35:02.294460 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:02.296917 12443 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:02.297508 12443 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:02.300514 12443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f75e8e65bf249959f673801b1e2f3ad: Generated new TSK 0
I20260928 06:35:02.300717 12443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:02.306250 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12355
W20260928 06:35:02.310390 12340 connection.cc:535] client connection to 127.0.0.1:38225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38225): Connection reset by peer (error 104)
2026-09-28T06:35:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12454](info): set up 2 sockets
Sep 28 06:35:02 dist-test-slave-38pg krb5kdc[12454](info): commencing operation
krb5kdc: starting...
W20260928 06:35:03.338820 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:03 dist-test-slave-38pg krb5kdc[12454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:03.465194 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:03.546936 12467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:03.547156 12467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:03.547187 12467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:03.548286 12467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:03.548321 12467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:03.548336 12467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:03.548348 12467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:03.548362 12467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:03.550302 12467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:03.550742 12467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:03.551082 12467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:03.556624 12473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:03.556653 12472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:03.556936 12467 server_base.cc:1034] running on GCE node
W20260928 06:35:03.556624 12475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:03.557231 12467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:03.557579 12467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:03.558771 12467 hybrid_clock.cc:648] HybridClock initialized: now 1790577303558745 us; error 40 us; skew 500 ppm
Sep 28 06:35:03 dist-test-slave-38pg krb5kdc[12454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:03.561472 12467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:03.562007 12467 webserver.cc:460] Webserver started at http://127.0.0.1:44863/ using document root <none> and password file <none>
I20260928 06:35:03.562261 12467 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:03.562331 12467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:03.562480 12467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:03.563586 12467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-38pg"
server_key: "8656fbc1347d9cf562d4e647b3817f6d"
server_key_iv: "9170a12e7b571aa9896df68893d2e351"
server_key_version: "encryptionkey@0"
I20260928 06:35:03.563979 12467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-38pg"
server_key: "8656fbc1347d9cf562d4e647b3817f6d"
server_key_iv: "9170a12e7b571aa9896df68893d2e351"
server_key_version: "encryptionkey@0"
I20260928 06:35:03.565333 12467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:35:03.566123 12482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:03.566296 12467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:03.566406 12467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3"
format_stamp: "Formatted at 2026-09-28 06:35:03 on dist-test-slave-38pg"
server_key: "8656fbc1347d9cf562d4e647b3817f6d"
server_key_iv: "9170a12e7b571aa9896df68893d2e351"
server_key_version: "encryptionkey@0"
I20260928 06:35:03.566483 12467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:03.584703 12467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:03.586941 12467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:03.587121 12467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:03.591468 12467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41153
I20260928 06:35:03.591503 12534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41153 every 8 connection(s)
I20260928 06:35:03.591874 12467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:03.592724 12535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:03.595036 12535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3: Bootstrap starting.
I20260928 06:35:03.595618 12535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:03.595917 12535 log.cc:824] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:03.596634 12535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3: No bootstrap required, opened a new log
I20260928 06:35:03.598410 12535 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260928 06:35:03.598551 12535 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:03.598582 12535 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca6d2d4bd8e45f4b7a965a775f8a9d3, State: Initialized, Role: FOLLOWER
I20260928 06:35:03.598763 12535 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260928 06:35:03.598835 12535 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:03.598857 12535 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:03.598924 12535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:03.599465 12535 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260928 06:35:03.599575 12535 leader_election.cc:302] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca6d2d4bd8e45f4b7a965a775f8a9d3; no voters:
I20260928 06:35:03.599757 12535 leader_election.cc:288] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:03.599875 12538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:03.600031 12535 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:03.600065 12538 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [term 1 LEADER]: Becoming Leader. State: Replica: eca6d2d4bd8e45f4b7a965a775f8a9d3, State: Running, Role: LEADER
I20260928 06:35:03.600268 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12467
I20260928 06:35:03.600289 12538 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260928 06:35:03.600392 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:03.600612 24622 external_mini_cluster.cc:1428] Setting key ac7cd1eb1e57b6df48fecc6d99ab5547
I20260928 06:35:03.601279 12539 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260928 06:35:03.601373 12540 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca6d2d4bd8e45f4b7a965a775f8a9d3. Latest consensus state: current_term: 1 leader_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca6d2d4bd8e45f4b7a965a775f8a9d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260928 06:35:03.601378 12539 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:03.601466 12540 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:03.601730 12548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:03.602535 12548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:03.604089 12548 catalog_manager.cc:1347] Generated new cluster ID: 86fd4377d03b44789a61d129513c70f3
I20260928 06:35:03.604143 12548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:03.607496 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:03.601827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52857 (local address 127.0.0.1:41153)
0928 06:35:03.602049 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:03.602059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:03.602094 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:03.602218 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:03.602224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:03.602296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:03.602488 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:03.602763 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.604202 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.605250 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.605905 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.606017 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:03.606787 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:03.606791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:03.606791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:03.607014 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:03.607053 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:03.607068 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:03.607158 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:03.607248 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:35:03.607275 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260928 06:35:03.608505 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:03.613731 12548 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:03.614009 12550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:03.610122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49325 (local address 127.0.0.1:41153)
0928 06:35:03.610156 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:03.610158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:03.610267 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:35:03.610323 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:03.610324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:03.610334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:03.610390 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:03.610523 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.611284 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.612467 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:03.612768 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:03.612837 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:03.613686 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:03.613687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:03.613687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:03.613700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:03.613716 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:03.613720 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:03.613775 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:03.613924 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:35:03.613938 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:03.614459 12548 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:03.618943 12548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eca6d2d4bd8e45f4b7a965a775f8a9d3: Generated new TSK 0
I20260928 06:35:03.619138 12548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:03.621758 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12467
W20260928 06:35:03.626077 12452 connection.cc:535] client connection to 127.0.0.1:41153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41153): Connection reset by peer (error 104)
2026-09-28T06:35:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:03 dist-test-slave-38pg krb5kdc[12566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:03 dist-test-slave-38pg krb5kdc[12566](info): set up 2 sockets
Sep 28 06:35:03 dist-test-slave-38pg krb5kdc[12566](info): commencing operation
krb5kdc: starting...
W20260928 06:35:04.655099 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:04 dist-test-slave-38pg krb5kdc[12566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:04.781620 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:04.865594 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:04.865862 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:04.865906 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:04.867050 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:04.867125 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:04.867148 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:04.867172 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:04.867197 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:04.868700 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:04.868983 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:04.869228 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:04.874475 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:04.874394 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:04.874450 12579 server_base.cc:1034] running on GCE node
W20260928 06:35:04.874395 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:04.874948 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:04.875252 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:04.876417 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790577304876405 us; error 45 us; skew 500 ppm
Sep 28 06:35:04 dist-test-slave-38pg krb5kdc[12566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:04.882552 12579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:04.882983 12579 webserver.cc:460] Webserver started at http://127.0.0.1:35489/ using document root <none> and password file <none>
I20260928 06:35:04.883184 12579 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:04.883229 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:04.883320 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:04.884400 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "85a8857518d94c6a88a79a186ff19f1e"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-38pg"
server_key: "7eec80dd0cf80e5604d8911f047dfcf8"
server_key_iv: "a14cbc4daf2b314015ba241f13e6d4d2"
server_key_version: "encryptionkey@0"
I20260928 06:35:04.884704 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "85a8857518d94c6a88a79a186ff19f1e"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-38pg"
server_key: "7eec80dd0cf80e5604d8911f047dfcf8"
server_key_iv: "a14cbc4daf2b314015ba241f13e6d4d2"
server_key_version: "encryptionkey@0"
I20260928 06:35:04.886101 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:04.886906 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:04.887106 12579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:04.887184 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "85a8857518d94c6a88a79a186ff19f1e"
format_stamp: "Formatted at 2026-09-28 06:35:04 on dist-test-slave-38pg"
server_key: "7eec80dd0cf80e5604d8911f047dfcf8"
server_key_iv: "a14cbc4daf2b314015ba241f13e6d4d2"
server_key_version: "encryptionkey@0"
I20260928 06:35:04.887279 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:04.890802 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:04.892719 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:04.892882 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:04.896878 12579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36535
I20260928 06:35:04.896910 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36535 every 8 connection(s)
I20260928 06:35:04.897243 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:04.898293 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:04.900533 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e: Bootstrap starting.
I20260928 06:35:04.901119 12647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:04.901409 12647 log.cc:824] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:04.902170 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e: No bootstrap required, opened a new log
I20260928 06:35:04.903762 12647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260928 06:35:04.903939 12647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:04.904002 12647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a8857518d94c6a88a79a186ff19f1e, State: Initialized, Role: FOLLOWER
I20260928 06:35:04.904177 12647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260928 06:35:04.904260 12647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:04.904302 12647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:04.904352 12647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:04.904919 12647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260928 06:35:04.905038 12647 leader_election.cc:302] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a8857518d94c6a88a79a186ff19f1e; no voters:
I20260928 06:35:04.905220 12647 leader_election.cc:288] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:04.905264 12650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:04.905472 12650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [term 1 LEADER]: Becoming Leader. State: Replica: 85a8857518d94c6a88a79a186ff19f1e, State: Running, Role: LEADER
I20260928 06:35:04.905494 12647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:04.905583 12650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } }
I20260928 06:35:04.905925 12651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a8857518d94c6a88a79a186ff19f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } } }
I20260928 06:35:04.905928 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a8857518d94c6a88a79a186ff19f1e. Latest consensus state: current_term: 1 leader_uuid: "85a8857518d94c6a88a79a186ff19f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a8857518d94c6a88a79a186ff19f1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36535 } } }
I20260928 06:35:04.906049 12651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:04.906114 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:04.906350 12654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:04.906620 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12579
I20260928 06:35:04.906755 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:04.906934 12654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:04.906965 24622 external_mini_cluster.cc:1428] Setting key 54c6aaf726d2247c2ef2bb352e57d6d2
I20260928 06:35:04.909162 12654 catalog_manager.cc:1347] Generated new cluster ID: 67eec6915ebd44b0bc5327363d32967c
I20260928 06:35:04.909250 12654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:04.909322 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.907600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46149 (local address 127.0.0.1:36535)
0928 06:35:04.907747 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.907756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.907791 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:35:04.907900 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.907906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.907930 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.908069 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.908800 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.908803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.908804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:04.908809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.908853 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:04.908868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.908938 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.909053 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.909097 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260928 06:35:04.910259 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:04.914345 12654 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:04.914496 12654 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:04.914884 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:04.911793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44359 (local address 127.0.0.1:36535)
0928 06:35:04.911829 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:04.911831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:04.913751 (+ 1920us) server_negotiation.cc:407] Connection header received
0928 06:35:04.913803 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:04.913804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:04.913810 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:04.913866 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:04.914625 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:04.914626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:04.914627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:04.914631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:04.914663 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:04.914669 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:04.914702 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:04.914807 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:35:04.914822 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:04.922201 12654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a8857518d94c6a88a79a186ff19f1e: Generated new TSK 0
I20260928 06:35:04.922401 12654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:04.924530 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12579
2026-09-28T06:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:04 dist-test-slave-38pg krb5kdc[12678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:04 dist-test-slave-38pg krb5kdc[12678](info): set up 2 sockets
Sep 28 06:35:04 dist-test-slave-38pg krb5kdc[12678](info): commencing operation
krb5kdc: starting...
W20260928 06:35:05.946660 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:06.064924 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:06.146412 12691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:06.146637 12691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:06.146664 12691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:06.147845 12691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:06.147882 12691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:06.147897 12691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:06.147909 12691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:06.147922 12691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:06.149341 12691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:06.149605 12691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:06.149808 12691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:06.154769 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:06.154858 12691 server_base.cc:1034] running on GCE node
W20260928 06:35:06.154778 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:06.154806 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:06.155241 12691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:06.155544 12691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:06.156718 12691 hybrid_clock.cc:648] HybridClock initialized: now 1790577306156694 us; error 57 us; skew 500 ppm
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:06.159255 12691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:06.159734 12691 webserver.cc:460] Webserver started at http://127.0.0.1:46577/ using document root <none> and password file <none>
I20260928 06:35:06.159960 12691 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:06.160027 12691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:06.160174 12691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:06.161245 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "a131d6beee11412596d3828677167499"
format_stamp: "Formatted at 2026-09-28 06:35:06 on dist-test-slave-38pg"
server_key: "4780e3dfceb9fdf66d02f995cb40ea69"
server_key_iv: "c6bca785ed0933df2e11da9be69a16a2"
server_key_version: "encryptionkey@0"
I20260928 06:35:06.161676 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "a131d6beee11412596d3828677167499"
format_stamp: "Formatted at 2026-09-28 06:35:06 on dist-test-slave-38pg"
server_key: "4780e3dfceb9fdf66d02f995cb40ea69"
server_key_iv: "c6bca785ed0933df2e11da9be69a16a2"
server_key_version: "encryptionkey@0"
I20260928 06:35:06.163028 12691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:06.163784 12706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:06.164001 12691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:06.164100 12691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "a131d6beee11412596d3828677167499"
format_stamp: "Formatted at 2026-09-28 06:35:06 on dist-test-slave-38pg"
server_key: "4780e3dfceb9fdf66d02f995cb40ea69"
server_key_iv: "c6bca785ed0933df2e11da9be69a16a2"
server_key_version: "encryptionkey@0"
I20260928 06:35:06.164186 12691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:06.176368 12691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:06.178786 12691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:06.178966 12691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:06.183262 12691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41937
I20260928 06:35:06.183303 12758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41937 every 8 connection(s)
I20260928 06:35:06.183660 12691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:06.184514 12759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:06.186931 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499: Bootstrap starting.
I20260928 06:35:06.187552 12759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:06.187862 12759 log.cc:824] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:06.188563 12759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499: No bootstrap required, opened a new log
I20260928 06:35:06.189895 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12691
I20260928 06:35:06.190027 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:06.190235 24622 external_mini_cluster.cc:1428] Setting key 6daac9f5e493d7dc4728d3bfe16ac043
I20260928 06:35:06.190770 12759 raft_consensus.cc:348] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260928 06:35:06.190986 12759 raft_consensus.cc:374] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:06.191040 12759 raft_consensus.cc:729] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a131d6beee11412596d3828677167499, State: Initialized, Role: FOLLOWER
I20260928 06:35:06.191176 12759 consensus_queue.cc:260] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260928 06:35:06.191262 12759 raft_consensus.cc:388] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:06.191306 12759 raft_consensus.cc:482] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:06.191355 12759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:06.192287 12759 raft_consensus.cc:504] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260928 06:35:06.192396 12759 leader_election.cc:302] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a131d6beee11412596d3828677167499; no voters:
I20260928 06:35:06.192546 12759 leader_election.cc:288] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:06.192610 12764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:06.192749 12764 raft_consensus.cc:686] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [term 1 LEADER]: Becoming Leader. State: Replica: a131d6beee11412596d3828677167499, State: Running, Role: LEADER
I20260928 06:35:06.192819 12759 sys_catalog.cc:564] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:06.192873 12764 consensus_queue.cc:237] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260928 06:35:06.193310 12765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a131d6beee11412596d3828677167499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260928 06:35:06.193286 12766 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a131d6beee11412596d3828677167499. Latest consensus state: current_term: 1 leader_uuid: "a131d6beee11412596d3828677167499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131d6beee11412596d3828677167499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260928 06:35:06.193377 12765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:06.193382 12766 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:06.193683 12771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:06.194722 12771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:06.196511 12771 catalog_manager.cc:1347] Generated new cluster ID: 4b037620e2194c41ab16c7955aeb11cc
I20260928 06:35:06.196569 12771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:35:06.201778 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:06.190769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59437 (local address 127.0.0.1:41937)
0928 06:35:06.190969 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:06.190977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:06.191021 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:35:06.191143 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:06.191151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:06.191206 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:06.191375 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:06.192099 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:06.193514 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:06.194595 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:06.195205 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:06.195313 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:06.198019 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:06.198023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:06.198023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:06.198036 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:06.199740 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:06.200130 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:06.200133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:06.200135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:06.200245 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:06.200471 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:06.200472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:06.200473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:06.200720 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:06.201348 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:06.201405 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:35:06.201515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260928 06:35:06.202855 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:06.206506 12771 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:06.207202 12771 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:06.212188 12763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:06.204240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43259 (local address 127.0.0.1:41937)
0928 06:35:06.204279 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:06.204281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:06.205807 (+ 1526us) server_negotiation.cc:407] Connection header received
0928 06:35:06.205881 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:06.205882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:06.205893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:06.205952 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:06.206658 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:06.207157 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:06.207911 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:06.208126 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:06.208210 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:06.209439 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:06.209441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:06.209441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:06.209448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:06.210980 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:06.211361 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:06.211362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:06.211362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:06.211425 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:06.211625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:06.211626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:06.211626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:06.211744 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:06.211876 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:06.211982 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:35:06.212075 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:06.216038 12771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a131d6beee11412596d3828677167499: Generated new TSK 0
I20260928 06:35:06.216254 12771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:06.220147 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12691
2026-09-28T06:35:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12790](info): set up 2 sockets
Sep 28 06:35:06 dist-test-slave-38pg krb5kdc[12790](info): commencing operation
krb5kdc: starting...
W20260928 06:35:07.246767 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:07 dist-test-slave-38pg krb5kdc[12790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:07.364965 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:07.445967 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:07.446175 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:07.446198 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:07.447284 12803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:07.447320 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:07.447335 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:07.447347 12803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:07.447359 12803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:07.448753 12803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:07.448951 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:07.449121 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:07.454073 12809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:07.454034 12811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:07.454075 12803 server_base.cc:1034] running on GCE node
W20260928 06:35:07.454046 12808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:07.454622 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:07.454932 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:07.456079 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790577307456071 us; error 32 us; skew 500 ppm
Sep 28 06:35:07 dist-test-slave-38pg krb5kdc[12790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:07.458624 12803 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:07.459035 12803 webserver.cc:460] Webserver started at http://127.0.0.1:42269/ using document root <none> and password file <none>
I20260928 06:35:07.459213 12803 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:07.459260 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:07.459348 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:07.460415 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "a8b1aee56bb14eb596010da3b4dc12a9"
format_stamp: "Formatted at 2026-09-28 06:35:07 on dist-test-slave-38pg"
server_key: "015564a1a6f2ecca79608187f70dc447"
server_key_iv: "fc3462288cadd1807983eeab54a51a46"
server_key_version: "encryptionkey@0"
I20260928 06:35:07.460721 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "a8b1aee56bb14eb596010da3b4dc12a9"
format_stamp: "Formatted at 2026-09-28 06:35:07 on dist-test-slave-38pg"
server_key: "015564a1a6f2ecca79608187f70dc447"
server_key_iv: "fc3462288cadd1807983eeab54a51a46"
server_key_version: "encryptionkey@0"
I20260928 06:35:07.462091 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:07.462798 12818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:07.462990 12803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:07.463083 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "a8b1aee56bb14eb596010da3b4dc12a9"
format_stamp: "Formatted at 2026-09-28 06:35:07 on dist-test-slave-38pg"
server_key: "015564a1a6f2ecca79608187f70dc447"
server_key_iv: "fc3462288cadd1807983eeab54a51a46"
server_key_version: "encryptionkey@0"
I20260928 06:35:07.463158 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:07.476022 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:07.478034 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:07.478193 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:07.482314 12803 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43067
I20260928 06:35:07.482374 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43067 every 8 connection(s)
I20260928 06:35:07.482726 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:07.483598 12871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:07.486063 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9: Bootstrap starting.
I20260928 06:35:07.486703 12871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:07.487017 12871 log.cc:824] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:07.487792 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9: No bootstrap required, opened a new log
I20260928 06:35:07.489506 12871 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 06:35:07.489687 12871 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:07.489753 12871 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8b1aee56bb14eb596010da3b4dc12a9, State: Initialized, Role: FOLLOWER
I20260928 06:35:07.489902 12871 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 06:35:07.489996 12871 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:07.490038 12871 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:07.490080 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12803
I20260928 06:35:07.490118 12871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:07.490211 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:07.490417 24622 external_mini_cluster.cc:1428] Setting key 2b7f4e8b8cd8c6e0534aabaddd27ee6d
I20260928 06:35:07.490897 12871 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 06:35:07.491058 12871 leader_election.cc:302] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8b1aee56bb14eb596010da3b4dc12a9; no voters:
I20260928 06:35:07.491237 12871 leader_election.cc:288] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:07.491294 12876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:07.491433 12876 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [term 1 LEADER]: Becoming Leader. State: Replica: a8b1aee56bb14eb596010da3b4dc12a9, State: Running, Role: LEADER
I20260928 06:35:07.491498 12871 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:07.491557 12876 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } }
I20260928 06:35:07.491932 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8b1aee56bb14eb596010da3b4dc12a9. Latest consensus state: current_term: 1 leader_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } } }
I20260928 06:35:07.491940 12878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8b1aee56bb14eb596010da3b4dc12a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43067 } } }
I20260928 06:35:07.492089 12878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:07.492082 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:07.492460 12880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:07.493291 12880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:07.495082 12880 catalog_manager.cc:1347] Generated new cluster ID: 3b185ba1b1d94ca6a693e6ae20eae64f
I20260928 06:35:07.495136 12880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:07.496901 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:07.490902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42849 (local address 127.0.0.1:43067)
0928 06:35:07.491120 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:07.491131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:07.491183 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:35:07.491330 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:07.491339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:07.491403 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:07.491838 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:07.491964 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:07.493669 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:07.494545 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:07.495207 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:07.495324 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:07.496099 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:07.496103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:07.496104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:07.496356 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:07.496388 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:07.496400 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:07.496486 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:07.496568 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:35:07.496736 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260928 06:35:07.497874 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:07.502492 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:07.499318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56497 (local address 127.0.0.1:43067)
0928 06:35:07.499358 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:07.499360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:07.499493 (+ 133us) server_negotiation.cc:407] Connection header received
0928 06:35:07.499572 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:07.499573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:07.499583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:07.499634 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:07.499760 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:07.500296 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:07.501079 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:07.501319 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:07.501399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:07.502198 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:07.502200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:07.502200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:07.502207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:07.502222 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:07.502226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:07.502286 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:07.502358 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:35:07.502384 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:07.510615 12880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:07.511215 12880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:07.515653 12880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8b1aee56bb14eb596010da3b4dc12a9: Generated new TSK 0
I20260928 06:35:07.515863 12880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:07.523820 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12803
2026-09-28T06:35:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:07 dist-test-slave-38pg krb5kdc[12902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:07 dist-test-slave-38pg krb5kdc[12902](info): set up 2 sockets
Sep 28 06:35:07 dist-test-slave-38pg krb5kdc[12902](info): commencing operation
krb5kdc: starting...
W20260928 06:35:08.571213 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:08 dist-test-slave-38pg krb5kdc[12902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:08.690459 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:08.771636 12915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:08.771852 12915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:08.771875 12915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:08.772964 12915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:08.773000 12915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:08.773015 12915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:08.773026 12915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:08.773038 12915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:08.775071 12915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:08.775280 12915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:08.775451 12915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:08.780452 12921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:08.780453 12920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:08.780617 12915 server_base.cc:1034] running on GCE node
W20260928 06:35:08.780462 12923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:08.781026 12915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:08.781345 12915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:08.782549 12915 hybrid_clock.cc:648] HybridClock initialized: now 1790577308782555 us; error 48 us; skew 500 ppm
Sep 28 06:35:08 dist-test-slave-38pg krb5kdc[12902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:08.785213 12915 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:08.785800 12915 webserver.cc:460] Webserver started at http://127.0.0.1:40437/ using document root <none> and password file <none>
I20260928 06:35:08.786049 12915 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:08.786121 12915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:08.786260 12915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:08.787405 12915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "ce5b473ae3cc48158e5c124087128862"
format_stamp: "Formatted at 2026-09-28 06:35:08 on dist-test-slave-38pg"
server_key: "38ccc38ebd9338332354da099652c16a"
server_key_iv: "15fb7e6579a8025cb87656298969ee80"
server_key_version: "encryptionkey@0"
I20260928 06:35:08.787791 12915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "ce5b473ae3cc48158e5c124087128862"
format_stamp: "Formatted at 2026-09-28 06:35:08 on dist-test-slave-38pg"
server_key: "38ccc38ebd9338332354da099652c16a"
server_key_iv: "15fb7e6579a8025cb87656298969ee80"
server_key_version: "encryptionkey@0"
I20260928 06:35:08.789222 12915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:08.790225 12930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:08.790452 12915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:08.790587 12915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "ce5b473ae3cc48158e5c124087128862"
format_stamp: "Formatted at 2026-09-28 06:35:08 on dist-test-slave-38pg"
server_key: "38ccc38ebd9338332354da099652c16a"
server_key_iv: "15fb7e6579a8025cb87656298969ee80"
server_key_version: "encryptionkey@0"
I20260928 06:35:08.790663 12915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:08.794138 12915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:08.796255 12915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:08.796429 12915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:08.800689 12915 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35591
I20260928 06:35:08.800755 12982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35591 every 8 connection(s)
I20260928 06:35:08.801098 12915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:08.802165 12983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:08.804518 12983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862: Bootstrap starting.
I20260928 06:35:08.804937 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 12915
I20260928 06:35:08.805068 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:08.805205 12983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:08.805241 24622 external_mini_cluster.cc:1428] Setting key 12e6e9a497b91219097ef023bc78eb40
I20260928 06:35:08.805493 12983 log.cc:824] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:08.806336 12983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862: No bootstrap required, opened a new log
I20260928 06:35:08.807960 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:08.805730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44651 (local address 127.0.0.1:35591)
0928 06:35:08.805921 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:08.805929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:08.805954 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:35:08.806631 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:08.806639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:08.806672 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:08.806824 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:08.807475 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:08.807478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:08.807478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:08.807482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:08.807529 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:08.807544 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:08.807641 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:08.807725 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:35:08.807764 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260928 06:35:08.808933 12983 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260928 06:35:08.809103 12983 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:08.809121 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:08.809137 12983 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5b473ae3cc48158e5c124087128862, State: Initialized, Role: FOLLOWER
I20260928 06:35:08.809233 12983 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260928 06:35:08.809291 12983 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:08.809314 12983 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:08.809341 12983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:08.809933 12983 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260928 06:35:08.810009 12983 leader_election.cc:302] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5b473ae3cc48158e5c124087128862; no voters:
I20260928 06:35:08.810163 12983 leader_election.cc:288] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:08.810189 12990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:08.810330 12990 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [term 1 LEADER]: Becoming Leader. State: Replica: ce5b473ae3cc48158e5c124087128862, State: Running, Role: LEADER
I20260928 06:35:08.810411 12983 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:08.810401 12990 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } }
I20260928 06:35:08.810726 12992 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5b473ae3cc48158e5c124087128862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } } }
I20260928 06:35:08.810712 12990 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5b473ae3cc48158e5c124087128862. Latest consensus state: current_term: 1 leader_uuid: "ce5b473ae3cc48158e5c124087128862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5b473ae3cc48158e5c124087128862" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35591 } } }
I20260928 06:35:08.811036 12992 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:08.811256 12990 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:08.811450 13002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:08.812091 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:08.810725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46655 (local address 127.0.0.1:35591)
0928 06:35:08.810769 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:08.810771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:08.810900 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:35:08.811004 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:08.811004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:08.811010 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:08.811041 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:08.811753 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:08.811755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:08.811755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:08.811759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:08.811774 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:08.811780 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:08.811827 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:08.811952 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:35:08.811966 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:08.811995 13002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:08.813926 13002 catalog_manager.cc:1347] Generated new cluster ID: ce9ef87220a94f7eab27b5fe584cb98f
I20260928 06:35:08.813997 13002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:08.821837 13002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:08.822038 13002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:08.831993 13002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5b473ae3cc48158e5c124087128862: Generated new TSK 0
I20260928 06:35:08.832226 13002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:08.834798 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 12915
2026-09-28T06:35:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:08 dist-test-slave-38pg krb5kdc[13014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:08 dist-test-slave-38pg krb5kdc[13014](info): set up 2 sockets
Sep 28 06:35:08 dist-test-slave-38pg krb5kdc[13014](info): commencing operation
krb5kdc: starting...
W20260928 06:35:09.870682 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:09 dist-test-slave-38pg krb5kdc[13014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:09.988793 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:10.071868 13027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:10.072072 13027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:10.072094 13027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:10.073261 13027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:10.073305 13027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:10.073319 13027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:10.073331 13027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:10.073343 13027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:10.074900 13027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:10.075155 13027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:10.075395 13027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:10.080502 13033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.080505 13032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:10.080533 13035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:10.080555 13027 server_base.cc:1034] running on GCE node
I20260928 06:35:10.080947 13027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:10.081279 13027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:10.082455 13027 hybrid_clock.cc:648] HybridClock initialized: now 1790577310082447 us; error 40 us; skew 500 ppm
Sep 28 06:35:10 dist-test-slave-38pg krb5kdc[13014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:10.093665 13027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:10.094161 13027 webserver.cc:460] Webserver started at http://127.0.0.1:38051/ using document root <none> and password file <none>
I20260928 06:35:10.094411 13027 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:10.094482 13027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:10.094631 13027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:10.095746 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "3e469396cad047de861446b9c68a46ff"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-38pg"
server_key: "fe446e2bff4120eda075af4a8624fd6e"
server_key_iv: "79544f738dd877a8fd03aa8a0ce3b84e"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.096249 13027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "3e469396cad047de861446b9c68a46ff"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-38pg"
server_key: "fe446e2bff4120eda075af4a8624fd6e"
server_key_iv: "79544f738dd877a8fd03aa8a0ce3b84e"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.097651 13027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:10.098399 13042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:10.098568 13027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:10.098677 13027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "3e469396cad047de861446b9c68a46ff"
format_stamp: "Formatted at 2026-09-28 06:35:10 on dist-test-slave-38pg"
server_key: "fe446e2bff4120eda075af4a8624fd6e"
server_key_iv: "79544f738dd877a8fd03aa8a0ce3b84e"
server_key_version: "encryptionkey@0"
I20260928 06:35:10.098752 13027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:10.124651 13027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:10.126842 13027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:10.127022 13027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:10.131313 13027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42447
I20260928 06:35:10.131367 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42447 every 8 connection(s)
I20260928 06:35:10.131748 13027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:10.132619 13095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:10.134064 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13027
I20260928 06:35:10.134207 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:10.134399 24622 external_mini_cluster.cc:1428] Setting key d46e4401d56b0ac78a5f8560ac0ed744
I20260928 06:35:10.135465 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff: Bootstrap starting.
I20260928 06:35:10.136179 13095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:10.136485 13095 log.cc:824] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:10.137197 13095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff: No bootstrap required, opened a new log
I20260928 06:35:10.139501 13095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260928 06:35:10.139683 13095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:10.139714 13095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e469396cad047de861446b9c68a46ff, State: Initialized, Role: FOLLOWER
I20260928 06:35:10.139865 13095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260928 06:35:10.139987 13095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:10.140033 13095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:10.140085 13095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:10.140856 13095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260928 06:35:10.140967 13095 leader_election.cc:302] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e469396cad047de861446b9c68a46ff; no voters:
I20260928 06:35:10.141152 13095 leader_election.cc:288] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:10.141184 13100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:10.141332 13100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [term 1 LEADER]: Becoming Leader. State: Replica: 3e469396cad047de861446b9c68a46ff, State: Running, Role: LEADER
I20260928 06:35:10.141479 13100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } }
I20260928 06:35:10.141549 13095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 06:35:10 dist-test-slave-38pg krb5kdc[13014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:35:10.141983 13102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e469396cad047de861446b9c68a46ff. Latest consensus state: current_term: 1 leader_uuid: "3e469396cad047de861446b9c68a46ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } } }
I20260928 06:35:10.142091 13102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:10.143083 13107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:10.143193 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e469396cad047de861446b9c68a46ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e469396cad047de861446b9c68a46ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42447 } } }
I20260928 06:35:10.143255 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:10.143963 13107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:10.145989 13107 catalog_manager.cc:1347] Generated new cluster ID: 64ef01c4fee34a1c98301c6f7fdf438b
I20260928 06:35:10.146051 13107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:10.146867 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.134982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:39809 (local address 127.0.0.1:42447)
0928 06:35:10.135181 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.135192 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.135226 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:35:10.135737 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.135747 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.135835 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.135993 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.136769 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.137855 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.138904 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.139289 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.139377 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.142995 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.143000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.143000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.143015 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.145203 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.145650 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.145653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.145654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.145754 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.145989 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.145990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.145990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.146282 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.146450 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.146584 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.146693 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260928 06:35:10.147800 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:10.155486 13098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:10.149133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:50583 (local address 127.0.0.1:42447)
0928 06:35:10.149168 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:10.149170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:10.149299 (+ 129us) server_negotiation.cc:407] Connection header received
0928 06:35:10.149369 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:10.149369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:10.149378 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:10.149430 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:10.149998 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.150484 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.151198 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:10.151426 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:10.151513 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:10.152726 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:10.152727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:10.152727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:10.152734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:10.154290 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.154652 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.154653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.154653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.154708 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:10.154950 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:10.154951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:10.154951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:10.155064 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:10.155181 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:10.155329 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:35:10.155396 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:10.155707 13107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:10.156401 13107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:10.161341 13107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e469396cad047de861446b9c68a46ff: Generated new TSK 0
I20260928 06:35:10.161511 13107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:10.166476 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13027
W20260928 06:35:10.170826 13012 connection.cc:535] client connection to 127.0.0.1:42447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42447): Connection reset by peer (error 104)
2026-09-28T06:35:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:10 dist-test-slave-38pg krb5kdc[13126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:10 dist-test-slave-38pg krb5kdc[13126](info): set up 2 sockets
Sep 28 06:35:10 dist-test-slave-38pg krb5kdc[13126](info): commencing operation
krb5kdc: starting...
W20260928 06:35:11.202821 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:11 dist-test-slave-38pg krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:11.324680 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:11.405263 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:11.405516 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:11.405571 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:11.406737 13139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:11.406811 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:11.406838 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:11.406872 13139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:11.406896 13139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:11.408349 13139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:11.408625 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:11.408861 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:35:11.413898 13139 server_base.cc:1034] running on GCE node
W20260928 06:35:11.413828 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:11.414011 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:11.413823 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:11.414296 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:11.414602 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:11.415772 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790577311415770 us; error 39 us; skew 500 ppm
Sep 28 06:35:11 dist-test-slave-38pg krb5kdc[13126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:11.418527 13139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:11.419023 13139 webserver.cc:460] Webserver started at http://127.0.0.1:37395/ using document root <none> and password file <none>
I20260928 06:35:11.419260 13139 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:11.419334 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:11.419473 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:11.420524 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "afddb24bb383400f81a432fd8e4b45b8"
format_stamp: "Formatted at 2026-09-28 06:35:11 on dist-test-slave-38pg"
server_key: "e2ec4345322d501a69973ff2f1873ee0"
server_key_iv: "2f61717e270aafc25c3567ae318e1cd6"
server_key_version: "encryptionkey@0"
I20260928 06:35:11.420898 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "afddb24bb383400f81a432fd8e4b45b8"
format_stamp: "Formatted at 2026-09-28 06:35:11 on dist-test-slave-38pg"
server_key: "e2ec4345322d501a69973ff2f1873ee0"
server_key_iv: "2f61717e270aafc25c3567ae318e1cd6"
server_key_version: "encryptionkey@0"
I20260928 06:35:11.422312 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:35:11.423017 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:11.423180 13139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:11.423286 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "afddb24bb383400f81a432fd8e4b45b8"
format_stamp: "Formatted at 2026-09-28 06:35:11 on dist-test-slave-38pg"
server_key: "e2ec4345322d501a69973ff2f1873ee0"
server_key_iv: "2f61717e270aafc25c3567ae318e1cd6"
server_key_version: "encryptionkey@0"
I20260928 06:35:11.423363 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:11.435187 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:11.437026 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:11.437194 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:11.441403 13139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39891
I20260928 06:35:11.441438 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39891 every 8 connection(s)
I20260928 06:35:11.441841 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:11.442670 13207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:11.444897 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: Bootstrap starting.
I20260928 06:35:11.445422 13207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:11.445736 13207 log.cc:824] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:11.446385 13207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: No bootstrap required, opened a new log
I20260928 06:35:11.447958 13207 raft_consensus.cc:348] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } }
I20260928 06:35:11.448091 13207 raft_consensus.cc:374] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:11.448113 13207 raft_consensus.cc:729] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afddb24bb383400f81a432fd8e4b45b8, State: Initialized, Role: FOLLOWER
I20260928 06:35:11.448246 13207 consensus_queue.cc:260] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } }
I20260928 06:35:11.448329 13207 raft_consensus.cc:388] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:11.448354 13207 raft_consensus.cc:482] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:11.448383 13207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:11.448845 13207 raft_consensus.cc:504] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } }
I20260928 06:35:11.448933 13207 leader_election.cc:302] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afddb24bb383400f81a432fd8e4b45b8; no voters:
I20260928 06:35:11.449048 13207 leader_election.cc:288] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:11.449131 13210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:11.449276 13210 raft_consensus.cc:686] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [term 1 LEADER]: Becoming Leader. State: Replica: afddb24bb383400f81a432fd8e4b45b8, State: Running, Role: LEADER
I20260928 06:35:11.449337 13207 sys_catalog.cc:564] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:11.449369 13210 consensus_queue.cc:237] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } }
I20260928 06:35:11.449779 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13139
I20260928 06:35:11.449929 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:11.450129 24622 external_mini_cluster.cc:1428] Setting key c8c6696f18077a3043bd15d8dbad14ca
I20260928 06:35:11.450690 13212 sys_catalog.cc:455] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afddb24bb383400f81a432fd8e4b45b8. Latest consensus state: current_term: 1 leader_uuid: "afddb24bb383400f81a432fd8e4b45b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } } }
I20260928 06:35:11.451000 13212 sys_catalog.cc:458] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:11.450691 13211 sys_catalog.cc:455] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afddb24bb383400f81a432fd8e4b45b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afddb24bb383400f81a432fd8e4b45b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39891 } } }
I20260928 06:35:11.451176 13211 sys_catalog.cc:458] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:11.451648 13226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:11.451709 13226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:11.451761 13228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:11.451920 13228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:11.453927 13228 catalog_manager.cc:1347] Generated new cluster ID: 126f7f8de92c40da9b5c53f4518d4e93
I20260928 06:35:11.453979 13228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:11.456542 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:11.450822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:57187 (local address 127.0.0.1:39891)
0928 06:35:11.450993 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:11.451001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:11.451168 (+ 167us) server_negotiation.cc:407] Connection header received
0928 06:35:11.451332 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:11.451340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:11.451403 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:11.451548 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:11.451815 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:11.453184 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:11.454228 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:11.454874 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:11.454990 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:11.455722 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:11.455726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:11.455726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:11.455950 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:11.455987 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:11.456003 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:11.456088 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:11.456180 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:35:11.456322 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260928 06:35:11.457440 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:11.466583 13228 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:11.467305 13228 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:11.469298 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:11.459072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:39905 (local address 127.0.0.1:39891)
0928 06:35:11.459107 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:11.459109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:11.466419 (+ 7310us) server_negotiation.cc:407] Connection header received
0928 06:35:11.466500 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:11.466501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:11.466512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:11.466569 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:11.466779 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:11.467258 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:11.468011 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:11.468237 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:11.468314 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:11.469020 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:11.469021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:11.469022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:11.469027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:11.469041 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:11.469046 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:11.469118 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:11.469184 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:35:11.469198 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:11.472214 13228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afddb24bb383400f81a432fd8e4b45b8: Generated new TSK 0
I20260928 06:35:11.472395 13228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:11.474203 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13139
W20260928 06:35:11.479638 13124 connection.cc:535] client connection to 127.0.0.1:39891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39891): Connection reset by peer (error 104)
2026-09-28T06:35:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:11 dist-test-slave-38pg krb5kdc[13238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:11 dist-test-slave-38pg krb5kdc[13238](info): set up 2 sockets
Sep 28 06:35:11 dist-test-slave-38pg krb5kdc[13238](info): commencing operation
krb5kdc: starting...
W20260928 06:35:12.510792 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:12 dist-test-slave-38pg krb5kdc[13238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:12.632162 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:12.712798 13251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:12.713004 13251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:12.713028 13251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:12.714181 13251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:12.714224 13251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:12.714238 13251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:12.714251 13251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:12.714263 13251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:12.715683 13251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:12.715878 13251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:12.716050 13251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:12.720938 13257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:12.720950 13259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.721055 13251 server_base.cc:1034] running on GCE node
W20260928 06:35:12.721139 13256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:12.721410 13251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:12.721773 13251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:12.722954 13251 hybrid_clock.cc:648] HybridClock initialized: now 1790577312722932 us; error 34 us; skew 500 ppm
Sep 28 06:35:12 dist-test-slave-38pg krb5kdc[13238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:12.725441 13251 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:12.725975 13251 webserver.cc:460] Webserver started at http://127.0.0.1:43325/ using document root <none> and password file <none>
I20260928 06:35:12.726225 13251 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:12.726295 13251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:12.726445 13251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:12.727532 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "e59b0ed6da6246e4bc329733ff83329f"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-38pg"
server_key: "435ef1065a56a37e928340bf4c9d1d5c"
server_key_iv: "2cd6aa8b7a7953a266bd424f32b78cdb"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.727901 13251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "e59b0ed6da6246e4bc329733ff83329f"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-38pg"
server_key: "435ef1065a56a37e928340bf4c9d1d5c"
server_key_iv: "2cd6aa8b7a7953a266bd424f32b78cdb"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.729234 13251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:12.730103 13266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:12.730278 13251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:12.730383 13251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "e59b0ed6da6246e4bc329733ff83329f"
format_stamp: "Formatted at 2026-09-28 06:35:12 on dist-test-slave-38pg"
server_key: "435ef1065a56a37e928340bf4c9d1d5c"
server_key_iv: "2cd6aa8b7a7953a266bd424f32b78cdb"
server_key_version: "encryptionkey@0"
I20260928 06:35:12.730460 13251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:12.733867 13251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:12.736207 13251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:12.736368 13251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:12.740401 13251 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35861
I20260928 06:35:12.740432 13318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35861 every 8 connection(s)
I20260928 06:35:12.740775 13251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:12.741786 13319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:12.744040 13319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f: Bootstrap starting.
I20260928 06:35:12.744629 13319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:12.744923 13319 log.cc:824] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:12.745712 13319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f: No bootstrap required, opened a new log
I20260928 06:35:12.747167 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13251
I20260928 06:35:12.747310 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:12.747493 24622 external_mini_cluster.cc:1428] Setting key 6974db2c707c8954b8a96a9566b73776
I20260928 06:35:12.747466 13319 raft_consensus.cc:348] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260928 06:35:12.747640 13319 raft_consensus.cc:374] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:12.747669 13319 raft_consensus.cc:729] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59b0ed6da6246e4bc329733ff83329f, State: Initialized, Role: FOLLOWER
I20260928 06:35:12.747838 13319 consensus_queue.cc:260] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260928 06:35:12.747933 13319 raft_consensus.cc:388] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:12.747967 13319 raft_consensus.cc:482] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:12.748014 13319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:12.749060 13319 raft_consensus.cc:504] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260928 06:35:12.749176 13319 leader_election.cc:302] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e59b0ed6da6246e4bc329733ff83329f; no voters:
I20260928 06:35:12.749329 13319 leader_election.cc:288] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:12.749370 13324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:12.749521 13324 raft_consensus.cc:686] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [term 1 LEADER]: Becoming Leader. State: Replica: e59b0ed6da6246e4bc329733ff83329f, State: Running, Role: LEADER
I20260928 06:35:12.749614 13319 sys_catalog.cc:564] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:12.749629 13324 consensus_queue.cc:237] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } }
I20260928 06:35:12.750097 13325 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e59b0ed6da6246e4bc329733ff83329f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260928 06:35:12.750185 13325 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:12.750372 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.748041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:34347 (local address 127.0.0.1:35861)
0928 06:35:12.748266 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.748275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.748507 (+ 232us) server_negotiation.cc:407] Connection header received
0928 06:35:12.748668 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.748676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.748714 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.748863 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.749777 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.749780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.749781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:12.749807 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.749853 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:12.749868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.749938 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.750074 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.750127 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":106,"threads_started":1}
I20260928 06:35:12.750892 13326 sys_catalog.cc:455] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e59b0ed6da6246e4bc329733ff83329f. Latest consensus state: current_term: 1 leader_uuid: "e59b0ed6da6246e4bc329733ff83329f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e59b0ed6da6246e4bc329733ff83329f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35861 } } }
I20260928 06:35:12.750943 13332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:12.751041 13326 sys_catalog.cc:458] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:12.751312 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:12.751581 13332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:12.753278 13332 catalog_manager.cc:1347] Generated new cluster ID: 62b1a93a4f5d43fdab2bc4d50cc5ad1e
I20260928 06:35:12.753327 13332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:12.754241 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:12.753035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:41135 (local address 127.0.0.1:35861)
0928 06:35:12.753071 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:12.753073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:12.753100 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:35:12.753228 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:12.753228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:12.753234 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:12.753288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:12.754011 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:12.754013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:12.754013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:12.754018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:12.754032 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:12.754036 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:12.754095 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:12.754151 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:35:12.754166 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:12.772404 13332 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:12.772600 13332 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:12.783910 13332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e59b0ed6da6246e4bc329733ff83329f: Generated new TSK 0
I20260928 06:35:12.784121 13332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:12.794711 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13251
2026-09-28T06:35:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:12 dist-test-slave-38pg krb5kdc[13350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:12 dist-test-slave-38pg krb5kdc[13350](info): set up 2 sockets
Sep 28 06:35:12 dist-test-slave-38pg krb5kdc[13350](info): commencing operation
krb5kdc: starting...
W20260928 06:35:13.822757 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:13 dist-test-slave-38pg krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:13.942616 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:14.024698 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:14.024978 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:14.025024 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:14.026268 13363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:14.026345 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:14.026369 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:14.026391 13363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:14.026422 13363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:14.027956 13363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:14.028247 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:14.028491 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:14.033566 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:14.033735 13363 server_base.cc:1034] running on GCE node
W20260928 06:35:14.033593 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:14.033665 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:14.034101 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:14.034368 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:14.035511 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790577314035452 us; error 74 us; skew 500 ppm
Sep 28 06:35:14 dist-test-slave-38pg krb5kdc[13350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:14.038139 13363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:14.038625 13363 webserver.cc:460] Webserver started at http://127.0.0.1:43557/ using document root <none> and password file <none>
I20260928 06:35:14.038885 13363 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:14.038935 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:14.039103 13363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:14.040201 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "9970832e1be3468680b1543a9cadd5f9"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-38pg"
server_key: "96891235176d9c26e8e2ad175e1b8faa"
server_key_iv: "242641be653ca55a0b22affbc1aa4614"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.040571 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "9970832e1be3468680b1543a9cadd5f9"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-38pg"
server_key: "96891235176d9c26e8e2ad175e1b8faa"
server_key_iv: "242641be653ca55a0b22affbc1aa4614"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.042016 13363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:14.042768 13378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:14.042958 13363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:14.043067 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "9970832e1be3468680b1543a9cadd5f9"
format_stamp: "Formatted at 2026-09-28 06:35:14 on dist-test-slave-38pg"
server_key: "96891235176d9c26e8e2ad175e1b8faa"
server_key_iv: "242641be653ca55a0b22affbc1aa4614"
server_key_version: "encryptionkey@0"
I20260928 06:35:14.043144 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:14.063990 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:14.066368 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:14.066557 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:14.070710 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32829
I20260928 06:35:14.070749 13430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32829 every 8 connection(s)
I20260928 06:35:14.071125 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:14.071990 13431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:14.074366 13431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: Bootstrap starting.
I20260928 06:35:14.074987 13431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:14.075299 13431 log.cc:824] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:14.076035 13431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: No bootstrap required, opened a new log
I20260928 06:35:14.077612 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13363
I20260928 06:35:14.077737 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:14.077963 24622 external_mini_cluster.cc:1428] Setting key bca3381f3d47b60cc2c8873d7431a580
I20260928 06:35:14.077879 13431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } }
I20260928 06:35:14.078011 13431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:14.078064 13431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9970832e1be3468680b1543a9cadd5f9, State: Initialized, Role: FOLLOWER
I20260928 06:35:14.078224 13431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } }
I20260928 06:35:14.078317 13431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:14.078353 13431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:14.078408 13431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:14.079547 13431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } }
I20260928 06:35:14.079675 13431 leader_election.cc:302] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9970832e1be3468680b1543a9cadd5f9; no voters:
I20260928 06:35:14.079860 13431 leader_election.cc:288] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:14.079885 13436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:14.080082 13436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [term 1 LEADER]: Becoming Leader. State: Replica: 9970832e1be3468680b1543a9cadd5f9, State: Running, Role: LEADER
I20260928 06:35:14.080240 13431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:14.080241 13436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } }
I20260928 06:35:14.080909 13438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9970832e1be3468680b1543a9cadd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } } }
I20260928 06:35:14.080996 13438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:14.081211 13437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9970832e1be3468680b1543a9cadd5f9. Latest consensus state: current_term: 1 leader_uuid: "9970832e1be3468680b1543a9cadd5f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9970832e1be3468680b1543a9cadd5f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32829 } } }
I20260928 06:35:14.081276 13437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9 [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:14.082269 13452 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:14.082337 13452 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:14.082657 13445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:14.082854 13445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:14.085634 13445 catalog_manager.cc:1347] Generated new cluster ID: fabb141d63004702bbbb6ed277f2ca9b
I20260928 06:35:14.085696 13445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:35:14 dist-test-slave-38pg krb5kdc[13350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:35:14.090381 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.078607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:44409 (local address 127.0.0.1:32829)
0928 06:35:14.078772 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:14.078780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:14.078809 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:35:14.078916 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:14.078921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:14.078966 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:14.079072 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:14.079963 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.082350 (+ 2387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.083436 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.083954 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.084080 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.086587 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:14.086595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:14.086596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:14.086607 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:14.088727 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.089146 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.089149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.089149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.089244 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.089490 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.089491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.089492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.089781 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:14.089977 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:14.090064 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:35:14.090167 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260928 06:35:14.091144 13445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:14.091404 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:14.091966 13445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:14.098989 13445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9970832e1be3468680b1543a9cadd5f9: Generated new TSK 0
I20260928 06:35:14.099217 13445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:14.099413 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:14.092860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:49825 (local address 127.0.0.1:32829)
0928 06:35:14.092898 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:14.092900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:14.093017 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:35:14.093091 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:14.093092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:14.093103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:14.093151 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:14.093713 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.094213 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.094928 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:14.095165 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:14.095250 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:14.096486 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:14.096487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:14.096488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:14.096495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:14.097993 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.098369 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.098370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.098370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.098434 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:14.098833 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:14.098835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:14.098835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:14.098951 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:14.099064 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:14.099226 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:35:14.099292 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:14.101457 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13363
W20260928 06:35:14.105394 13348 connection.cc:535] client connection to 127.0.0.1:32829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32829): Connection reset by peer (error 104)
2026-09-28T06:35:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:14 dist-test-slave-38pg krb5kdc[13462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:14 dist-test-slave-38pg krb5kdc[13462](info): set up 2 sockets
Sep 28 06:35:14 dist-test-slave-38pg krb5kdc[13462](info): commencing operation
krb5kdc: starting...
W20260928 06:35:15.130826 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:15 dist-test-slave-38pg krb5kdc[13462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:15.252846 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:15.333949 13475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:15.334156 13475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:15.334178 13475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:15.335322 13475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:15.335359 13475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:15.335374 13475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:15.335386 13475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:15.335399 13475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:15.336831 13475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:15.337074 13475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:15.337316 13475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:15.342415 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:15.342406 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:15.342406 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:15.342475 13475 server_base.cc:1034] running on GCE node
I20260928 06:35:15.342888 13475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:15.343178 13475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:15.344337 13475 hybrid_clock.cc:648] HybridClock initialized: now 1790577315344316 us; error 38 us; skew 500 ppm
Sep 28 06:35:15 dist-test-slave-38pg krb5kdc[13462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:15.353214 13475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:15.353825 13475 webserver.cc:460] Webserver started at http://127.0.0.1:41375/ using document root <none> and password file <none>
I20260928 06:35:15.354070 13475 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:15.354141 13475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:15.354305 13475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:15.355443 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "35ff47c046e34ab6880aac33a278f0b7"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-38pg"
server_key: "08a227de5f75b729699d67152601044c"
server_key_iv: "1f9dda78c038373bcdc7dd959609f900"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.355825 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "35ff47c046e34ab6880aac33a278f0b7"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-38pg"
server_key: "08a227de5f75b729699d67152601044c"
server_key_iv: "1f9dda78c038373bcdc7dd959609f900"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.357247 13475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:15.358160 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:15.358386 13475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:15.358490 13475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "35ff47c046e34ab6880aac33a278f0b7"
format_stamp: "Formatted at 2026-09-28 06:35:15 on dist-test-slave-38pg"
server_key: "08a227de5f75b729699d67152601044c"
server_key_iv: "1f9dda78c038373bcdc7dd959609f900"
server_key_version: "encryptionkey@0"
I20260928 06:35:15.358582 13475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:15.373440 13475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:15.375409 13475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:15.375588 13475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:15.379848 13475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35781
I20260928 06:35:15.379886 13542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35781 every 8 connection(s)
I20260928 06:35:15.380249 13475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:15.381147 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:15.383522 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7: Bootstrap starting.
I20260928 06:35:15.384110 13543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:15.384418 13543 log.cc:824] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:15.385113 13543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7: No bootstrap required, opened a new log
I20260928 06:35:15.386760 13543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260928 06:35:15.386904 13543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:15.386929 13543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ff47c046e34ab6880aac33a278f0b7, State: Initialized, Role: FOLLOWER
I20260928 06:35:15.387095 13543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260928 06:35:15.387157 13543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:15.387210 13543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:15.387261 13543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:15.387892 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13475
I20260928 06:35:15.387880 13543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260928 06:35:15.388015 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:15.388001 13543 leader_election.cc:302] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35ff47c046e34ab6880aac33a278f0b7; no voters:
I20260928 06:35:15.388171 13543 leader_election.cc:288] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:15.388238 24622 external_mini_cluster.cc:1428] Setting key 22880df4755f9d0343b74d3f0c2b2e66
I20260928 06:35:15.388216 13546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:15.388387 13546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [term 1 LEADER]: Becoming Leader. State: Replica: 35ff47c046e34ab6880aac33a278f0b7, State: Running, Role: LEADER
I20260928 06:35:15.388490 13543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:15.388528 13546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } }
I20260928 06:35:15.389281 13548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35ff47c046e34ab6880aac33a278f0b7. Latest consensus state: current_term: 1 leader_uuid: "35ff47c046e34ab6880aac33a278f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } } }
I20260928 06:35:15.389389 13548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:15.389612 13557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:15.389756 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35ff47c046e34ab6880aac33a278f0b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ff47c046e34ab6880aac33a278f0b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35781 } } }
I20260928 06:35:15.389808 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:15.390113 13557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:15.391682 13557 catalog_manager.cc:1347] Generated new cluster ID: 5f695bf5d52b4743ada82fe68377e137
I20260928 06:35:15.391732 13557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:15.394330 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.388812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:58551 (local address 127.0.0.1:35781)
0928 06:35:15.389014 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.389026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.389066 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:35:15.389200 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.389207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.389276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.389712 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.389857 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.391057 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.392185 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.392758 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.392868 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.393634 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.393638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.393638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:15.393846 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.393883 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:15.393896 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.393971 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.394096 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.394121 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260928 06:35:15.395371 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:15.405165 13550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:15.396855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:53273 (local address 127.0.0.1:35781)
0928 06:35:15.396889 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:15.396891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:15.401750 (+ 4859us) server_negotiation.cc:407] Connection header received
0928 06:35:15.401820 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:15.401821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:15.401833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:15.401892 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:15.402025 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.402745 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.403776 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:15.404105 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:15.404165 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:15.404911 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:15.404912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:15.404912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:15.404916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:15.404931 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:15.404936 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:15.404986 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:15.405076 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:35:15.405089 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:15.408361 13557 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:15.408874 13557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:15.414129 13557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35ff47c046e34ab6880aac33a278f0b7: Generated new TSK 0
I20260928 06:35:15.414284 13557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:15.416515 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13475
W20260928 06:35:15.420915 13460 connection.cc:535] client connection to 127.0.0.1:35781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35781): Connection reset by peer (error 104)
2026-09-28T06:35:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:15 dist-test-slave-38pg krb5kdc[13574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:15 dist-test-slave-38pg krb5kdc[13574](info): set up 2 sockets
Sep 28 06:35:15 dist-test-slave-38pg krb5kdc[13574](info): commencing operation
krb5kdc: starting...
W20260928 06:35:16.447016 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:16 dist-test-slave-38pg krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:35:16.562537 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:16.644079 13587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:16.644294 13587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:16.644316 13587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:16.645406 13587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:16.645440 13587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:16.645454 13587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:16.645467 13587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:16.645479 13587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:16.647050 13587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46333
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:16.647246 13587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:16.647416 13587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:16.652370 13592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:16.652336 13595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:16.652405 13587 server_base.cc:1034] running on GCE node
W20260928 06:35:16.652340 13593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:16.652833 13587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:16.653160 13587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:16.654313 13587 hybrid_clock.cc:648] HybridClock initialized: now 1790577316654283 us; error 42 us; skew 500 ppm
Sep 28 06:35:16 dist-test-slave-38pg krb5kdc[13574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:16.656832 13587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:35:16.657277 13587 webserver.cc:460] Webserver started at http://127.0.0.1:38123/ using document root <none> and password file <none>
I20260928 06:35:16.657559 13587 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:16.657639 13587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:16.657796 13587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:16.658867 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "30e49dc30a844a12b0a295e9260a7a73"
format_stamp: "Formatted at 2026-09-28 06:35:16 on dist-test-slave-38pg"
server_key: "52ac8de1d32f32ba124f370adc58ce32"
server_key_iv: "4f86748b78d343ca93ad41aad7d731dd"
server_key_version: "encryptionkey@0"
I20260928 06:35:16.659236 13587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "30e49dc30a844a12b0a295e9260a7a73"
format_stamp: "Formatted at 2026-09-28 06:35:16 on dist-test-slave-38pg"
server_key: "52ac8de1d32f32ba124f370adc58ce32"
server_key_iv: "4f86748b78d343ca93ad41aad7d731dd"
server_key_version: "encryptionkey@0"
I20260928 06:35:16.660585 13587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:16.661342 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:16.661512 13587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:16.661633 13587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "30e49dc30a844a12b0a295e9260a7a73"
format_stamp: "Formatted at 2026-09-28 06:35:16 on dist-test-slave-38pg"
server_key: "52ac8de1d32f32ba124f370adc58ce32"
server_key_iv: "4f86748b78d343ca93ad41aad7d731dd"
server_key_version: "encryptionkey@0"
I20260928 06:35:16.661720 13587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:16.664947 13587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:16.666900 13587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:16.667060 13587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:16.671134 13587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46333
I20260928 06:35:16.671206 13654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46333 every 8 connection(s)
I20260928 06:35:16.671543 13587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:16.672560 13655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:16.674837 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73: Bootstrap starting.
I20260928 06:35:16.675364 13655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:16.675595 13655 log.cc:824] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:16.676214 13655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73: No bootstrap required, opened a new log
I20260928 06:35:16.677090 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13587
I20260928 06:35:16.677207 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:16.677404 24622 external_mini_cluster.cc:1428] Setting key 7886a7cbf905189038651d20f672e418
I20260928 06:35:16.678444 13655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 06:35:16.678618 13655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:16.678659 13655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e49dc30a844a12b0a295e9260a7a73, State: Initialized, Role: FOLLOWER
I20260928 06:35:16.678855 13655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 06:35:16.678951 13655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:16.679000 13655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:16.679092 13655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:16.679839 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.678002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:46601 (local address 127.0.0.1:46333)
0928 06:35:16.678166 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.678174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.678198 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:35:16.678425 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.678431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.678456 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.678569 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:16.679330 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:16.679334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:16.679335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:16.679339 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:16.679390 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:16.679407 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:16.679485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.679545 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.679598 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 06:35:16.680289 13655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 06:35:16.680394 13655 leader_election.cc:302] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e49dc30a844a12b0a295e9260a7a73; no voters:
I20260928 06:35:16.680610 13655 leader_election.cc:288] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:16.680631 13660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:16.680738 13660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [term 1 LEADER]: Becoming Leader. State: Replica: 30e49dc30a844a12b0a295e9260a7a73, State: Running, Role: LEADER
I20260928 06:35:16.680830 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:16.680867 13655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:16.680867 13660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260928 06:35:16.681275 13661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30e49dc30a844a12b0a295e9260a7a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260928 06:35:16.681284 13662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e49dc30a844a12b0a295e9260a7a73. Latest consensus state: current_term: 1 leader_uuid: "30e49dc30a844a12b0a295e9260a7a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e49dc30a844a12b0a295e9260a7a73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260928 06:35:16.681380 13661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:16.681404 13662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:16.681715 13668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:16.682344 13668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:16.683831 13668 catalog_manager.cc:1347] Generated new cluster ID: ed734de588b14832b8e8e5d44f5f804e
I20260928 06:35:16.683888 13668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:16.690965 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:16.682560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.45:35055 (local address 127.0.0.1:46333)
0928 06:35:16.682592 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:16.682594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:16.689959 (+ 7365us) server_negotiation.cc:407] Connection header received
0928 06:35:16.690012 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:16.690013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:16.690018 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:16.690076 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:16.690740 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:16.690741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:16.690741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:16.690746 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:16.690764 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:16.690769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:16.690833 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:16.690885 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 06:35:16.690899 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:35:16.708628 13668 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:16.708829 13668 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:16.716184 13668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e49dc30a844a12b0a295e9260a7a73: Generated new TSK 0
I20260928 06:35:16.716388 13668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:16.724455 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13587
2026-09-28T06:35:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:16 dist-test-slave-38pg krb5kdc[13686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:16 dist-test-slave-38pg krb5kdc[13686](info): set up 2 sockets
Sep 28 06:35:16 dist-test-slave-38pg krb5kdc[13686](info): commencing operation
krb5kdc: starting...
W20260928 06:35:17.826786 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:17 dist-test-slave-38pg krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:17.944248 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43003
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:35339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:18.025908 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:18.026176 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:18.026218 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:18.027403 13699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:18.027477 13699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:18.027501 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:18.027523 13699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:18.027552 13699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:18.029081 13699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:35339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:18.029376 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:18.029637 13699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:18.034580 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.034565 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:18.034554 13707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:18.034807 13699 server_base.cc:1034] running on GCE node
I20260928 06:35:18.035029 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:18.035274 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:18.036423 13699 hybrid_clock.cc:648] HybridClock initialized: now 1790577318036410 us; error 29 us; skew 500 ppm
Sep 28 06:35:18 dist-test-slave-38pg krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577318, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:18.039047 13699 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:18.039444 13699 webserver.cc:460] Webserver started at http://127.24.11.190:38981/ using document root <none> and password file <none>
I20260928 06:35:18.039647 13699 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:18.039688 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:18.039778 13699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:18.040784 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "6880bc7917d54cb39ddc143125140746"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-38pg"
server_key: "4c318e7c8c5c51b328d758b2d221a552"
server_key_iv: "94af1a10dcca44db44f181ccbdf1143e"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.041088 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "6880bc7917d54cb39ddc143125140746"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-38pg"
server_key: "4c318e7c8c5c51b328d758b2d221a552"
server_key_iv: "94af1a10dcca44db44f181ccbdf1143e"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.042487 13699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:18.043193 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:18.043407 13699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:18.043500 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "6880bc7917d54cb39ddc143125140746"
format_stamp: "Formatted at 2026-09-28 06:35:18 on dist-test-slave-38pg"
server_key: "4c318e7c8c5c51b328d758b2d221a552"
server_key_iv: "94af1a10dcca44db44f181ccbdf1143e"
server_key_version: "encryptionkey@0"
I20260928 06:35:18.043577 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:18.080900 13699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:18.084014 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:18.084196 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:18.088724 13699 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43003
I20260928 06:35:18.088786 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43003 every 8 connection(s)
I20260928 06:35:18.089171 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:18.089848 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13699
I20260928 06:35:18.089985 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:18.090196 24622 external_mini_cluster.cc:1428] Setting key 661ba456a6767b9902fd7298f80b8f78
I20260928 06:35:18.090334 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:18.093654 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746: Bootstrap starting.
I20260928 06:35:18.094353 13767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:18.094707 13767 log.cc:824] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:18.095554 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746: No bootstrap required, opened a new log
Sep 28 06:35:18 dist-test-slave-38pg krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:18.097937 13767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } }
I20260928 06:35:18.098100 13767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:18.098124 13767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6880bc7917d54cb39ddc143125140746, State: Initialized, Role: FOLLOWER
I20260928 06:35:18.098266 13767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } }
I20260928 06:35:18.098359 13767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:18.098390 13767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:18.098431 13767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:18.099107 13767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } }
I20260928 06:35:18.099210 13767 leader_election.cc:302] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6880bc7917d54cb39ddc143125140746; no voters:
I20260928 06:35:18.099438 13767 leader_election.cc:288] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:18.099501 13772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:18.099721 13767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:18.099731 13772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [term 1 LEADER]: Becoming Leader. State: Replica: 6880bc7917d54cb39ddc143125140746, State: Running, Role: LEADER
I20260928 06:35:18.099886 13772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } }
I20260928 06:35:18.100272 13772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6880bc7917d54cb39ddc143125140746. Latest consensus state: current_term: 1 leader_uuid: "6880bc7917d54cb39ddc143125140746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } } }
I20260928 06:35:18.101649 13772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:18.101886 13787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:18.100807 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6880bc7917d54cb39ddc143125140746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6880bc7917d54cb39ddc143125140746" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43003 } } }
I20260928 06:35:18.102066 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:18.102432 13787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:18.102480 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.090758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37913 (local address 127.24.11.190:43003)
0928 06:35:18.090937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.090947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.090985 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:18.091474 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.091481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.091529 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.091646 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.092485 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.093406 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.094498 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.094986 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.095074 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.097983 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.097987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.097987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.097998 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.100221 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.100685 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.100688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.100688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.100817 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.101123 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.101124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.101125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.101446 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.101650 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.102257 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.102338 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260928 06:35:18.103706 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:18.104234 13787 catalog_manager.cc:1347] Generated new cluster ID: 861d246417b342faa4387c82f6900f84
I20260928 06:35:18.104282 13787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:18.112676 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:18.105210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46453 (local address 127.24.11.190:43003)
0928 06:35:18.105245 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:18.105247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:18.105292 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:35:18.105357 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:18.105358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:18.105367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:18.105436 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:18.106020 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.106787 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.107761 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:18.108108 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:18.108194 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:18.109768 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:18.109770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:18.109770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:18.109777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:18.111433 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.111841 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.111843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.111844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.111910 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:18.112139 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:18.112140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:18.112140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:18.112277 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:18.112386 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:18.112514 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:35:18.112581 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:18.114187 13787 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:18.114856 13787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:18.118498 13787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6880bc7917d54cb39ddc143125140746: Generated new TSK 0
I20260928 06:35:18.118688 13787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:18.123562 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13699
W20260928 06:35:18.128451 13684 connection.cc:535] client connection to 127.24.11.190:43003 recv error: Network error: failed to read from TLS socket (remote: 127.24.11.190:43003): Connection reset by peer (error 104)
2026-09-28T06:35:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:18 dist-test-slave-38pg krb5kdc[13798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:18 dist-test-slave-38pg krb5kdc[13798](info): set up 2 sockets
Sep 28 06:35:18 dist-test-slave-38pg krb5kdc[13798](info): commencing operation
krb5kdc: starting...
W20260928 06:35:19.166584 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:19 dist-test-slave-38pg krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:19.282064 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43689
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:37997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:19.361773 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:19.362007 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:19.362046 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:19.363198 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:19.363267 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:19.363294 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:19.363314 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:19.363345 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:19.364833 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:37997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:43689
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:19.365113 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:19.365350 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:19.370302 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:19.370318 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:19.370352 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:19.370355 13811 server_base.cc:1034] running on GCE node
I20260928 06:35:19.370787 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:19.371073 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:19.372236 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790577319372209 us; error 43 us; skew 500 ppm
Sep 28 06:35:19 dist-test-slave-38pg krb5kdc[13798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577319, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:19.374804 13811 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:19.375254 13811 webserver.cc:460] Webserver started at http://127.24.11.190:37779/ using document root <none> and password file <none>
I20260928 06:35:19.375496 13811 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:19.375543 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:19.375699 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:19.376775 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "47bcb04588d14fe99e1439aa244cbdec"
format_stamp: "Formatted at 2026-09-28 06:35:19 on dist-test-slave-38pg"
server_key: "7c21112253cd45d04970857cb3a830ec"
server_key_iv: "a9b430345011629871a1aa2dfb694ce0"
server_key_version: "encryptionkey@0"
I20260928 06:35:19.377130 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "47bcb04588d14fe99e1439aa244cbdec"
format_stamp: "Formatted at 2026-09-28 06:35:19 on dist-test-slave-38pg"
server_key: "7c21112253cd45d04970857cb3a830ec"
server_key_iv: "a9b430345011629871a1aa2dfb694ce0"
server_key_version: "encryptionkey@0"
I20260928 06:35:19.378571 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:19.379387 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:19.379683 13811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:19.379806 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "47bcb04588d14fe99e1439aa244cbdec"
format_stamp: "Formatted at 2026-09-28 06:35:19 on dist-test-slave-38pg"
server_key: "7c21112253cd45d04970857cb3a830ec"
server_key_iv: "a9b430345011629871a1aa2dfb694ce0"
server_key_version: "encryptionkey@0"
I20260928 06:35:19.379882 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:19.392616 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:19.394672 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:19.394838 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:19.398846 13811 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:43689
I20260928 06:35:19.398882 13878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:43689 every 8 connection(s)
I20260928 06:35:19.399243 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:19.400061 13879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:19.402328 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec: Bootstrap starting.
I20260928 06:35:19.402894 13879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:19.403182 13879 log.cc:824] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:19.403872 13879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec: No bootstrap required, opened a new log
I20260928 06:35:19.405421 13879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } }
I20260928 06:35:19.405571 13879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:19.405620 13879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcb04588d14fe99e1439aa244cbdec, State: Initialized, Role: FOLLOWER
I20260928 06:35:19.405778 13879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } }
I20260928 06:35:19.405864 13879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:19.405915 13879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:19.405963 13879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:19.406738 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13811
I20260928 06:35:19.406765 13879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } }
I20260928 06:35:19.406836 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:19.406888 13879 leader_election.cc:302] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47bcb04588d14fe99e1439aa244cbdec; no voters:
I20260928 06:35:19.407047 24622 external_mini_cluster.cc:1428] Setting key 560b3b0879e76ffa635aaf5699821ac6
I20260928 06:35:19.407083 13879 leader_election.cc:288] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:19.407106 13882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:19.407327 13879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:19.407646 13882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [term 1 LEADER]: Becoming Leader. State: Replica: 47bcb04588d14fe99e1439aa244cbdec, State: Running, Role: LEADER
I20260928 06:35:19.407785 13882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } }
I20260928 06:35:19.408246 13885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47bcb04588d14fe99e1439aa244cbdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } } }
I20260928 06:35:19.408339 13885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:19.408551 13883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47bcb04588d14fe99e1439aa244cbdec. Latest consensus state: current_term: 1 leader_uuid: "47bcb04588d14fe99e1439aa244cbdec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcb04588d14fe99e1439aa244cbdec" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 43689 } } }
I20260928 06:35:19.408660 13883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:19.408847 13894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:19.409356 13894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:19.410903 13894 catalog_manager.cc:1347] Generated new cluster ID: e0829f3e47cc4d948a87fcf4f240ffee
I20260928 06:35:19.410986 13894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:19.412858 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.407595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37951 (local address 127.24.11.190:43689)
0928 06:35:19.407790 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.407798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.407824 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:19.407935 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.407942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.407991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.408361 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:19.408484 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.409905 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.410934 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.411402 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.411538 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.412232 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:19.412235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:19.412236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:19.412410 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:19.412438 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:19.412449 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:19.412541 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.412618 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.412644 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260928 06:35:19.413777 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:19.420436 13886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:19.415084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49053 (local address 127.24.11.190:43689)
0928 06:35:19.415120 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:19.415122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:19.417745 (+ 2623us) server_negotiation.cc:407] Connection header received
0928 06:35:19.417816 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:19.417817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:19.417827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:19.417870 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:19.418026 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.418496 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.419221 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:19.419449 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:19.419532 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:19.420199 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:19.420201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:19.420201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:19.420205 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:19.420218 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:19.420222 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:19.420274 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:19.420333 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:35:19.420344 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:19.482092 13894 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:19.482800 13894 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:19.494220 13894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47bcb04588d14fe99e1439aa244cbdec: Generated new TSK 0
I20260928 06:35:19.494462 13894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:19.502880 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13811
W20260928 06:35:19.509011 13796 connection.cc:535] client connection to 127.24.11.190:43689 recv error: Network error: failed to read from TLS socket (remote: 127.24.11.190:43689): Connection reset by peer (error 104)
2026-09-28T06:35:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:19 dist-test-slave-38pg krb5kdc[13910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:19 dist-test-slave-38pg krb5kdc[13910](info): set up 2 sockets
Sep 28 06:35:19 dist-test-slave-38pg krb5kdc[13910](info): commencing operation
krb5kdc: starting...
W20260928 06:35:20.550906 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:20 dist-test-slave-38pg krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:20.672883 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37897
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:20.754541 13923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:20.754842 13923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:20.754882 13923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:20.756152 13923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:20.756222 13923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:20.756246 13923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:20.756268 13923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:20.756296 13923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:20.757866 13923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:20.758158 13923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:20.758395 13923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:20.763540 13928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:20.763595 13931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:20.763549 13929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:20.763918 13923 server_base.cc:1034] running on GCE node
I20260928 06:35:20.764195 13923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:20.764496 13923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:20.765720 13923 hybrid_clock.cc:648] HybridClock initialized: now 1790577320765714 us; error 62 us; skew 500 ppm
Sep 28 06:35:20 dist-test-slave-38pg krb5kdc[13910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577320, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:20.771869 13923 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:20.772455 13923 webserver.cc:460] Webserver started at http://127.24.11.190:43395/ using document root <none> and password file <none>
I20260928 06:35:20.772701 13923 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:20.772773 13923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:20.772930 13923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:20.774118 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "8462efb027e646c58d0018518a7f92a5"
format_stamp: "Formatted at 2026-09-28 06:35:20 on dist-test-slave-38pg"
server_key: "7c009c2c8c46dc19cb814d80141b5ead"
server_key_iv: "f9b5785f458def46ad3a6fd8a0a7e898"
server_key_version: "encryptionkey@0"
I20260928 06:35:20.774520 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "8462efb027e646c58d0018518a7f92a5"
format_stamp: "Formatted at 2026-09-28 06:35:20 on dist-test-slave-38pg"
server_key: "7c009c2c8c46dc19cb814d80141b5ead"
server_key_iv: "f9b5785f458def46ad3a6fd8a0a7e898"
server_key_version: "encryptionkey@0"
I20260928 06:35:20.776031 13923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:20.776911 13938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:20.777175 13923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:20.777344 13923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "8462efb027e646c58d0018518a7f92a5"
format_stamp: "Formatted at 2026-09-28 06:35:20 on dist-test-slave-38pg"
server_key: "7c009c2c8c46dc19cb814d80141b5ead"
server_key_iv: "f9b5785f458def46ad3a6fd8a0a7e898"
server_key_version: "encryptionkey@0"
I20260928 06:35:20.777462 13923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:20.781013 13923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:20.783325 13923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:20.783505 13923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:20.787859 13923 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:37897
I20260928 06:35:20.787889 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:37897 every 8 connection(s)
I20260928 06:35:20.788240 13923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:20.789283 13991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:20.791795 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5: Bootstrap starting.
I20260928 06:35:20.792476 13991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:20.792804 13991 log.cc:824] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:20.793596 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5: No bootstrap required, opened a new log
I20260928 06:35:20.795284 13991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } }
I20260928 06:35:20.795424 13991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:20.795447 13991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8462efb027e646c58d0018518a7f92a5, State: Initialized, Role: FOLLOWER
I20260928 06:35:20.795614 13991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } }
I20260928 06:35:20.795677 13991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:20.795719 13991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:20.795766 13991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:20.796640 13991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } }
I20260928 06:35:20.796736 13991 leader_election.cc:302] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8462efb027e646c58d0018518a7f92a5; no voters:
I20260928 06:35:20.796921 13991 leader_election.cc:288] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:20.797008 13994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:20.797173 13994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [term 1 LEADER]: Becoming Leader. State: Replica: 8462efb027e646c58d0018518a7f92a5, State: Running, Role: LEADER
I20260928 06:35:20.797201 13991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:20.797266 13994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } }
I20260928 06:35:20.797664 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8462efb027e646c58d0018518a7f92a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } } }
I20260928 06:35:20.797677 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8462efb027e646c58d0018518a7f92a5. Latest consensus state: current_term: 1 leader_uuid: "8462efb027e646c58d0018518a7f92a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8462efb027e646c58d0018518a7f92a5" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37897 } } }
I20260928 06:35:20.797736 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 13923
I20260928 06:35:20.797762 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:20.797773 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:20.797873 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:20.797986 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:20.798089 24622 external_mini_cluster.cc:1428] Setting key 562ab606a66cf633e1ab67aa3e317487
I20260928 06:35:20.798520 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:20.800055 14002 catalog_manager.cc:1347] Generated new cluster ID: 513c3e5958ad46ccb73b922708f69399
I20260928 06:35:20.800105 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:20.800690 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:20.798697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33815 (local address 127.24.11.190:37897)
0928 06:35:20.798970 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:20.798979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:20.799020 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:35:20.799192 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:20.799206 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:20.799261 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:20.799429 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:20.800239 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:20.800248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:20.800249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:20.800253 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:20.800287 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:20.800298 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:20.800369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:20.800451 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:35:20.800497 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260928 06:35:20.801496 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:20.804051 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:20.802884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41321 (local address 127.24.11.190:37897)
0928 06:35:20.802927 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:20.802929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:20.803053 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:35:20.803124 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:20.803124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:20.803144 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:20.803206 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:20.803824 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:20.803825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:20.803826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:20.803830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:20.803845 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:20.803849 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:20.803906 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:20.803974 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:35:20.803987 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:20.813594 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:20.813776 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:20.817385 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8462efb027e646c58d0018518a7f92a5: Generated new TSK 0
I20260928 06:35:20.817642 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:20.821780 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 13923
2026-09-28T06:35:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:20 dist-test-slave-38pg krb5kdc[14022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:20 dist-test-slave-38pg krb5kdc[14022](info): set up 2 sockets
Sep 28 06:35:20 dist-test-slave-38pg krb5kdc[14022](info): commencing operation
krb5kdc: starting...
W20260928 06:35:21.858809 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:21 dist-test-slave-38pg krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:21.980755 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36837
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:22.069260 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:22.069484 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:22.069510 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:22.070847 14035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:22.070894 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:22.070910 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:22.070924 14035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:22.070976 14035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:22.072589 14035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:22.072844 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:22.073101 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:22.078104 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:22.078217 14035 server_base.cc:1034] running on GCE node
W20260928 06:35:22.078118 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:22.078114 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:22.078574 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:22.078914 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:22.080060 14035 hybrid_clock.cc:648] HybridClock initialized: now 1790577322080050 us; error 30 us; skew 500 ppm
Sep 28 06:35:22 dist-test-slave-38pg krb5kdc[14022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577322, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:22.082697 14035 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:22.083200 14035 webserver.cc:460] Webserver started at http://127.24.11.190:40205/ using document root <none> and password file <none>
I20260928 06:35:22.083461 14035 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:22.083532 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:22.083680 14035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:22.084780 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "9a1e37b679364c1bb01039446dbaa9df"
format_stamp: "Formatted at 2026-09-28 06:35:22 on dist-test-slave-38pg"
server_key: "429f46d099d5b4d3e56011ca055667d0"
server_key_iv: "b0ab6b9d3e554d06bd1791685469387c"
server_key_version: "encryptionkey@0"
I20260928 06:35:22.085156 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "9a1e37b679364c1bb01039446dbaa9df"
format_stamp: "Formatted at 2026-09-28 06:35:22 on dist-test-slave-38pg"
server_key: "429f46d099d5b4d3e56011ca055667d0"
server_key_iv: "b0ab6b9d3e554d06bd1791685469387c"
server_key_version: "encryptionkey@0"
I20260928 06:35:22.086634 14035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:22.087435 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:22.087610 14035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:35:22.087719 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "9a1e37b679364c1bb01039446dbaa9df"
format_stamp: "Formatted at 2026-09-28 06:35:22 on dist-test-slave-38pg"
server_key: "429f46d099d5b4d3e56011ca055667d0"
server_key_iv: "b0ab6b9d3e554d06bd1791685469387c"
server_key_version: "encryptionkey@0"
I20260928 06:35:22.087807 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:22.096897 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:22.098973 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:22.099140 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:22.103466 14035 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:36837
I20260928 06:35:22.103514 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:36837 every 8 connection(s)
I20260928 06:35:22.103868 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:22.104729 14103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:22.105912 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14035
I20260928 06:35:22.106029 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:22.106209 24622 external_mini_cluster.cc:1428] Setting key 68b56cfab3ff9ef9cf4a3be02f7c4dfa
I20260928 06:35:22.108386 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df: Bootstrap starting.
I20260928 06:35:22.109007 14103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:22.109308 14103 log.cc:824] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:22.110018 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df: No bootstrap required, opened a new log
I20260928 06:35:22.111850 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } }
I20260928 06:35:22.111982 14103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:22.112004 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a1e37b679364c1bb01039446dbaa9df, State: Initialized, Role: FOLLOWER
I20260928 06:35:22.112144 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } }
I20260928 06:35:22.112210 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:22.112232 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:22.112283 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 0 FOLLOWER]: Advancing to term 1
Sep 28 06:35:22 dist-test-slave-38pg krb5kdc[14022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:22.112866 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } }
I20260928 06:35:22.112963 14103 leader_election.cc:302] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a1e37b679364c1bb01039446dbaa9df; no voters:
I20260928 06:35:22.113168 14103 leader_election.cc:288] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:22.113206 14108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:22.113322 14108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [term 1 LEADER]: Becoming Leader. State: Replica: 9a1e37b679364c1bb01039446dbaa9df, State: Running, Role: LEADER
I20260928 06:35:22.113402 14108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } }
I20260928 06:35:22.113435 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:22.113718 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a1e37b679364c1bb01039446dbaa9df. Latest consensus state: current_term: 1 leader_uuid: "9a1e37b679364c1bb01039446dbaa9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } } }
I20260928 06:35:22.113786 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:22.113950 14109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a1e37b679364c1bb01039446dbaa9df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a1e37b679364c1bb01039446dbaa9df" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 36837 } } }
I20260928 06:35:22.114059 14109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:22.114133 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:22.114969 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:22.117194 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:22.106733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33419 (local address 127.24.11.190:36837)
0928 06:35:22.106941 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:22.106949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:22.106975 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:35:22.107392 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:22.107399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:22.107445 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:22.107560 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:22.108365 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:22.109372 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:22.110269 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:22.110670 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:22.110789 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:22.113347 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:22.113351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:22.113351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:22.113365 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:22.115666 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:22.116063 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:22.116065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:22.116066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:22.116147 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:22.116361 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:22.116361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:22.116362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:22.116577 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:22.116693 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:22.116850 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 06:35:22.117018 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260928 06:35:22.118214 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:22.122675 14112 catalog_manager.cc:1347] Generated new cluster ID: 6345f554ffda48fdbb63cd518aa5168a
I20260928 06:35:22.122735 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:22.125959 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:22.119429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57065 (local address 127.24.11.190:36837)
0928 06:35:22.119468 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:22.119471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:22.119518 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:35:22.119591 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:22.119592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:22.119601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:22.119654 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:22.120198 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:22.120680 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:22.121363 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:22.121662 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:22.121749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:22.123221 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:22.123222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:22.123222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:22.123229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:22.124818 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:22.125165 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:22.125165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:22.125166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:22.125211 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:22.125375 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:22.125375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:22.125376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:22.125484 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:22.125596 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:22.125797 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:35:22.125872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:35:22.137010 14112 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:22.137651 14112 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:22.142277 14112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a1e37b679364c1bb01039446dbaa9df: Generated new TSK 0
I20260928 06:35:22.142467 14112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:22.150143 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14035
2026-09-28T06:35:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:22 dist-test-slave-38pg krb5kdc[14134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:22 dist-test-slave-38pg krb5kdc[14134](info): set up 2 sockets
Sep 28 06:35:22 dist-test-slave-38pg krb5kdc[14134](info): commencing operation
krb5kdc: starting...
W20260928 06:35:23.182827 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:23 dist-test-slave-38pg krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:23.301054 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34643
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:45893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:34643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:23.382966 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:23.383232 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:23.383283 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:23.384519 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:23.384595 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:23.384619 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:23.384640 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:23.384672 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:23.386299 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:45893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:34643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:23.386607 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:23.386847 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:23.392236 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:23.392241 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:23.392253 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:23.392524 14147 server_base.cc:1034] running on GCE node
I20260928 06:35:23.392795 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:23.393133 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:23.394305 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790577323394291 us; error 45 us; skew 500 ppm
Sep 28 06:35:23 dist-test-slave-38pg krb5kdc[14134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577323, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:23.396865 14147 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:23.397336 14147 webserver.cc:460] Webserver started at http://127.24.11.190:44277/ using document root <none> and password file <none>
I20260928 06:35:23.397596 14147 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:23.397646 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:23.397740 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:23.398767 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "013c074cf9244eaa8a07401fd1070e8e"
format_stamp: "Formatted at 2026-09-28 06:35:23 on dist-test-slave-38pg"
server_key: "2854797994b68cf2c46936171c5ccc62"
server_key_iv: "4885d4d5a7544d125a8d47b154559b4a"
server_key_version: "encryptionkey@0"
I20260928 06:35:23.399083 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "013c074cf9244eaa8a07401fd1070e8e"
format_stamp: "Formatted at 2026-09-28 06:35:23 on dist-test-slave-38pg"
server_key: "2854797994b68cf2c46936171c5ccc62"
server_key_iv: "4885d4d5a7544d125a8d47b154559b4a"
server_key_version: "encryptionkey@0"
I20260928 06:35:23.400357 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:23.401047 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:23.401225 14147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:23.401332 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "013c074cf9244eaa8a07401fd1070e8e"
format_stamp: "Formatted at 2026-09-28 06:35:23 on dist-test-slave-38pg"
server_key: "2854797994b68cf2c46936171c5ccc62"
server_key_iv: "4885d4d5a7544d125a8d47b154559b4a"
server_key_version: "encryptionkey@0"
I20260928 06:35:23.401412 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:23.425617 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:23.428051 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:23.428225 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:23.432331 14147 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:34643
I20260928 06:35:23.432366 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:34643 every 8 connection(s)
I20260928 06:35:23.432714 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:23.433607 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:23.435817 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e: Bootstrap starting.
I20260928 06:35:23.436210 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14147
I20260928 06:35:23.436326 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:23.436517 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:23.436527 24622 external_mini_cluster.cc:1428] Setting key 027e5353be9ca6d8ee431c3d3676e648
I20260928 06:35:23.436853 14215 log.cc:824] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:23.437640 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e: No bootstrap required, opened a new log
I20260928 06:35:23.440104 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } }
I20260928 06:35:23.440234 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:23.440258 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013c074cf9244eaa8a07401fd1070e8e, State: Initialized, Role: FOLLOWER
I20260928 06:35:23.440387 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } }
I20260928 06:35:23.440457 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:23.440478 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:23.440542 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:23.441228 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } }
I20260928 06:35:23.441332 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 013c074cf9244eaa8a07401fd1070e8e; no voters:
I20260928 06:35:23.441484 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:23.441568 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:23.441725 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [term 1 LEADER]: Becoming Leader. State: Replica: 013c074cf9244eaa8a07401fd1070e8e, State: Running, Role: LEADER
I20260928 06:35:23.441797 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:23.442250 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } }
I20260928 06:35:23.442771 14222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 013c074cf9244eaa8a07401fd1070e8e. Latest consensus state: current_term: 1 leader_uuid: "013c074cf9244eaa8a07401fd1070e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } } }
I20260928 06:35:23.442857 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "013c074cf9244eaa8a07401fd1070e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "013c074cf9244eaa8a07401fd1070e8e" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 34643 } } }
I20260928 06:35:23.442967 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:23.442867 14222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:23.443279 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:23.437039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52689 (local address 127.24.11.190:34643)
0928 06:35:23.437236 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:23.437245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:23.437311 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:35:23.437930 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:23.437939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:23.437999 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:23.438141 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:23.438338 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:23.439770 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:23.440718 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:23.441350 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:23.441473 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:23.442422 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:23.442426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:23.442426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:23.442650 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:23.442701 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:23.442716 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:23.442762 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:23.442918 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:35:23.442989 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260928 06:35:23.443288 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:23.444075 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:23.445266 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:23.446602 14229 catalog_manager.cc:1347] Generated new cluster ID: 4057549c5c5c40c2be0637affa9842b4
I20260928 06:35:23.446894 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:23.450428 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:23.446726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60747 (local address 127.24.11.190:34643)
0928 06:35:23.446762 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:23.446764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:23.446889 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:35:23.447030 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:23.447031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:23.447041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:23.447070 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:23.447259 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:23.447991 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:23.449052 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:23.449347 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:23.449387 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:23.450176 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:23.450177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:23.450177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:23.450182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:23.450195 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:23.450199 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:23.450227 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:23.450331 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:35:23.450354 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:23.458415 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:23.458905 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:23.464414 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 013c074cf9244eaa8a07401fd1070e8e: Generated new TSK 0
I20260928 06:35:23.464586 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:23.469658 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14147
2026-09-28T06:35:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:23 dist-test-slave-38pg krb5kdc[14246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:23 dist-test-slave-38pg krb5kdc[14246](info): set up 2 sockets
Sep 28 06:35:23 dist-test-slave-38pg krb5kdc[14246](info): commencing operation
krb5kdc: starting...
W20260928 06:35:24.506714 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:24 dist-test-slave-38pg krb5kdc[14246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:24.624891 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:45405
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:45405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:24.707170 14259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:24.707377 14259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:24.707401 14259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:24.708545 14259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:24.708592 14259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:24.708606 14259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:24.708618 14259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:24.708631 14259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:24.710225 14259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:45405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:24.710480 14259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:24.710704 14259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:24.715685 14264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:24.715817 14259 server_base.cc:1034] running on GCE node
W20260928 06:35:24.715723 14267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:24.715723 14265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:24.716171 14259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:24.716485 14259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:24.717594 14259 hybrid_clock.cc:648] HybridClock initialized: now 1790577324717588 us; error 38 us; skew 500 ppm
Sep 28 06:35:24 dist-test-slave-38pg krb5kdc[14246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577324, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:24.720021 14259 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:24.720494 14259 webserver.cc:460] Webserver started at http://127.24.11.190:45287/ using document root <none> and password file <none>
I20260928 06:35:24.720716 14259 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:24.720782 14259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:24.720963 14259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:24.722052 14259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "8ec7b5b7c4264c26af23a52896aa20fd"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-38pg"
server_key: "b0df53eb24e58cc6d6199a0c45446717"
server_key_iv: "8e11c2b97a5640edbee9ca6be2cf4e84"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.722355 14259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "8ec7b5b7c4264c26af23a52896aa20fd"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-38pg"
server_key: "b0df53eb24e58cc6d6199a0c45446717"
server_key_iv: "8e11c2b97a5640edbee9ca6be2cf4e84"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.723557 14259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:24.724241 14274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:24.724406 14259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:24.724505 14259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "8ec7b5b7c4264c26af23a52896aa20fd"
format_stamp: "Formatted at 2026-09-28 06:35:24 on dist-test-slave-38pg"
server_key: "b0df53eb24e58cc6d6199a0c45446717"
server_key_iv: "8e11c2b97a5640edbee9ca6be2cf4e84"
server_key_version: "encryptionkey@0"
I20260928 06:35:24.724577 14259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:24.727838 14259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:24.729763 14259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:24.729910 14259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:24.734007 14259 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:45405
I20260928 06:35:24.734042 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:45405 every 8 connection(s)
I20260928 06:35:24.734381 14259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:24.735347 14327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:24.737704 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: Bootstrap starting.
I20260928 06:35:24.738283 14327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:24.738601 14327 log.cc:824] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:24.739354 14327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: No bootstrap required, opened a new log
I20260928 06:35:24.739569 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14259
I20260928 06:35:24.739701 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:24.739882 24622 external_mini_cluster.cc:1428] Setting key 9af579c10ecfa6ecfc33b0266f6e4d3d
I20260928 06:35:24.741905 14327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } }
I20260928 06:35:24.742079 14327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:35:24.742126 14327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec7b5b7c4264c26af23a52896aa20fd, State: Initialized, Role: FOLLOWER
I20260928 06:35:24.742298 14327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } }
I20260928 06:35:24.742724 14327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:35:24.742801 14327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:35:24.742843 14327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:24.742904 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.740399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53773 (local address 127.24.11.190:45405)
0928 06:35:24.740590 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:24.740599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:24.740633 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:35:24.741379 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:24.741388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:24.741423 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:24.741609 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:24.742333 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:24.742336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:24.742336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:24.742340 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:24.742378 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:24.742388 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:24.742445 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:24.742548 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:35:24.742580 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260928 06:35:24.743608 14327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } }
I20260928 06:35:24.743731 14327 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec7b5b7c4264c26af23a52896aa20fd; no voters:
I20260928 06:35:24.743886 14327 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:35:24.743886 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:24.744055 14333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:24.744159 14327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:24.744176 14333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [term 1 LEADER]: Becoming Leader. State: Replica: 8ec7b5b7c4264c26af23a52896aa20fd, State: Running, Role: LEADER
I20260928 06:35:24.745689 14333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } }
I20260928 06:35:24.746089 14332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec7b5b7c4264c26af23a52896aa20fd. Latest consensus state: current_term: 1 leader_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } } }
I20260928 06:35:24.746186 14332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:24.746306 14334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec7b5b7c4264c26af23a52896aa20fd" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 45405 } } }
I20260928 06:35:24.746492 14334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:24.746802 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:24.745370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34979 (local address 127.24.11.190:45405)
0928 06:35:24.745411 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:24.745414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:24.745641 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:35:24.745685 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:24.745685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:24.745690 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:24.745748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:24.746552 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:24.746553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:24.746554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:35:24.746558 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:24.746571 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:35:24.746577 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:24.746608 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:24.746722 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:35:24.746736 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:35:24.746805 14351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:24.746912 14351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:24.747124 14352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:24.747357 14352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:24.748917 14352 catalog_manager.cc:1347] Generated new cluster ID: 86858b969f654b9f84da76cf4c7de879
I20260928 06:35:24.748987 14352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:24.756506 14352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:24.756675 14352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:24.762624 14352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec7b5b7c4264c26af23a52896aa20fd: Generated new TSK 0
I20260928 06:35:24.762817 14352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:24.767144 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14259
2026-09-28T06:35:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23779 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:24 dist-test-slave-38pg krb5kdc[14358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:24 dist-test-slave-38pg krb5kdc[14358](info): set up 2 sockets
Sep 28 06:35:24 dist-test-slave-38pg krb5kdc[14358](info): commencing operation
krb5kdc: starting...
W20260928 06:35:25.806809 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:25 dist-test-slave-38pg krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:25.924178 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37121
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:26.005623 14371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:26.005901 14371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:26.005932 14371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:26.005981 14371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:26.007167 14371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:26.007246 14371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:26.007272 14371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:26.007303 14371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:26.007329 14371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:26.008872 14371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:26.009158 14371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:26.009421 14371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:26.014770 14379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.014755 14376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.014732 14377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:26.014796 14371 server_base.cc:1034] running on GCE node
I20260928 06:35:26.015211 14371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:26.015492 14371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:26.016647 14371 hybrid_clock.cc:648] HybridClock initialized: now 1790577326016629 us; error 40 us; skew 500 ppm
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:26.024479 14371 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:26.025010 14371 webserver.cc:460] Webserver started at http://127.24.11.190:39489/ using document root <none> and password file <none>
I20260928 06:35:26.025275 14371 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:26.025344 14371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:26.025492 14371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:26.026651 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "eed6e2baffc97f2175ab4dbf48a8d916"
server_key_iv: "2c1c89a851df16ea71b677c040c66bcb"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.027025 14371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "eed6e2baffc97f2175ab4dbf48a8d916"
server_key_iv: "2c1c89a851df16ea71b677c040c66bcb"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.028388 14371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:26.029145 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:26.029313 14371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:26.029428 14371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "eed6e2baffc97f2175ab4dbf48a8d916"
server_key_iv: "2c1c89a851df16ea71b677c040c66bcb"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.029503 14371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:26.037498 14371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:26.039939 14371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:26.040108 14371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:26.044379 14371 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:37121
I20260928 06:35:26.044414 14438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:37121 every 8 connection(s)
I20260928 06:35:26.044778 14371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:26.045907 14439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:26.047422 14439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.048799 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14371
I20260928 06:35:26.048938 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:26.049163 24622 external_mini_cluster.cc:1428] Setting key c4fcc890d5e3550b5f8167956282f33c
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.11.189@KRBTEST.COM; defaulting to no policy
I20260928 06:35:26.058904 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.048037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37121 (local address 127.0.0.1:50460)
0928 06:35:26.048308 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.048338 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.048953 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.049252 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.049255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.049702 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.049941 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.049946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.051013 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.051016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.051754 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.051756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.052287 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.052289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.052352 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.052935 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.052944 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.054966 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.057020 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.057028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.057031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.057321 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.057620 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.057621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.057622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.057763 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.058267 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.058271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.058336 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.058464 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.058710 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":96,"thread_start_us":106,"threads_started":1}
I20260928 06:35:26.058908 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.048092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.24.11.190:37121)
0928 06:35:26.048252 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.048260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.048441 (+ 181us) server_negotiation.cc:407] Connection header received
0928 06:35:26.049112 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.049114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.049131 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.049210 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.050040 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.050893 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.051851 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.052162 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.052297 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.055070 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.055078 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.055078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.055086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.056903 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.057410 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.057414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.057414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.057494 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.057897 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.057898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.057899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.058196 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.058291 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.058499 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.058646 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.189@KRBTEST.COM" created.
I20260928 06:35:26.059934 14439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.060310 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.060194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45464)
0928 06:35:26.060232 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.060282 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 06:35:26.060483 14389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
I20260928 06:35:26.060947 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.060859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45468)
0928 06:35:26.060905 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.060930 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:35:26.061069 14439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
I20260928 06:35:26.076092 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:39433
--webserver_interface=127.24.11.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:26.110293 14439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } attempt: 1
I20260928 06:35:26.110829 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.110692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45470)
0928 06:35:26.110759 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.110800 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:35:26.111222 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.111132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45478)
0928 06:35:26.111174 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.111202 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:35:26.111332 14439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
W20260928 06:35:26.158249 14446 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:26.158479 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:26.158499 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:26.158519 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:26.159713 14446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:26.159754 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:26.159768 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:26.159814 14446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:26.159845 14446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:26.161343 14446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:39433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.11.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:26.161623 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:26.161875 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:26.166767 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.166795 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.166760 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:26.166970 14446 server_base.cc:1034] running on GCE node
I20260928 06:35:26.167287 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:26.167608 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:26.168787 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790577326168769 us; error 26 us; skew 500 ppm
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:26.171550 14446 init.cc:377] Logged in from keytab as kudu/127.24.11.189@KRBTEST.COM (short username kudu)
I20260928 06:35:26.171953 14446 webserver.cc:460] Webserver started at http://127.24.11.189:36967/ using document root <none> and password file <none>
I20260928 06:35:26.172151 14446 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:26.172191 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:26.172279 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:26.173290 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data/instance:
uuid: "76477bdbb9f24af596fa7b92cd2fe013"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "877524a65c0811d87b675abf3e19e2df"
server_key_iv: "e8ca886552f175cd35ca079b081b10f3"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.173669 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal/instance:
uuid: "76477bdbb9f24af596fa7b92cd2fe013"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "877524a65c0811d87b675abf3e19e2df"
server_key_iv: "e8ca886552f175cd35ca079b081b10f3"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.174960 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:26.175662 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:26.175865 14446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:26.175968 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "76477bdbb9f24af596fa7b92cd2fe013"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "877524a65c0811d87b675abf3e19e2df"
server_key_iv: "e8ca886552f175cd35ca079b081b10f3"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.176043 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:26.179486 14439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } attempt: 2
I20260928 06:35:26.179880 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.179760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45480)
0928 06:35:26.179816 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.179852 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:35:26.180220 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.180140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45488)
0928 06:35:26.180180 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.180204 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:35:26.180322 14439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39433: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39433: connect: Connection refused (error 111)
I20260928 06:35:26.189160 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:26.191650 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:26.191816 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:26.195782 14446 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.189:39433
I20260928 06:35:26.195830 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.189:39433 every 8 connection(s)
I20260928 06:35:26.196216 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
I20260928 06:35:26.197213 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:26.198807 14514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.201035 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14446
I20260928 06:35:26.201149 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-1/wal/instance
I20260928 06:35:26.201332 24622 external_mini_cluster.cc:1428] Setting key ad5f0e8c76223bf2514d70951433c8f5
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:26.208348 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.199508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.24.11.190:37121)
0928 06:35:26.199575 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.199577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.199801 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:35:26.200225 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.200226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.200234 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.200298 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.200906 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.201662 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.202619 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.202846 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.202922 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.205063 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.205064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.205065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.205074 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.206888 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.207308 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.207310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.207310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.207381 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.207622 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.207623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.207624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.207772 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.207861 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.208144 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.208231 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:35:26.208389 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.199482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37121 (local address 127.0.0.1:50474)
0928 06:35:26.199675 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.199718 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.200147 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.200348 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.200352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.200589 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.200777 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.200780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.201749 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.201750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.202524 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.202526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.202968 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.202969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.203082 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.203632 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.203640 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.204973 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.206975 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.206982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.206984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.207229 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.207453 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.207453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.207454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.207553 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.207865 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.207871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.208001 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.208097 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.208229 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260928 06:35:26.209022 14514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.11.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.188@KRBTEST.COM" created.
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:26.217306 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.209326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45492)
0928 06:35:26.209389 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.209402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.209434 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.209811 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.209812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.209948 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.209992 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.209993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.210943 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.210945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.211524 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.211525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.212078 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.212079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.212145 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.212597 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.212602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.214063 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.216197 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.216198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.216198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.216362 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.216585 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.216585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.216586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.216635 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.217051 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.217052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.217093 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.217158 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.217234 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:35:26.217456 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.209384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.24.11.189:39433)
0928 06:35:26.209604 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.209608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.209636 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:35:26.209690 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.209691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.209702 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.209808 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.210068 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.210854 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.211614 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.211989 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.212078 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.214145 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.214152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.214152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.214158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.216113 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.216443 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.216447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.216448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.216512 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.216731 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.216732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.216733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.216952 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.217051 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.217221 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.217313 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":102,"threads_started":1}
I20260928 06:35:26.218161 14514 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.218552 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.218412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42890)
0928 06:35:26.218465 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.218521 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:35:26.218707 14464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
I20260928 06:35:26.219180 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.219074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42896)
0928 06:35:26.219126 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.219157 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:35:26.219297 14514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38695: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
I20260928 06:35:26.229964 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:38695
--webserver_interface=127.24.11.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:26.268540 14514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } attempt: 1
I20260928 06:35:26.269153 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.269008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42908)
0928 06:35:26.269069 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.269118 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:35:26.269603 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.269447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42916)
0928 06:35:26.269500 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.269568 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 06:35:26.269730 14514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38695: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
I20260928 06:35:26.271486 14439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } attempt: 3
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:26.278821 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.271869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.24.11.189:39433)
0928 06:35:26.271931 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.271934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.271945 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:26.272021 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.272022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.272034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.272090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.272409 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.272931 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.273745 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.273981 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.274056 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.276163 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.276165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.276166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.276177 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.277627 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.277981 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.277982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.277983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.278044 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.278254 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.278255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.278256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.278379 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.278468 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.278633 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.278717 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:35:26.278827 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.271808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45500)
0928 06:35:26.271861 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.271880 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.271953 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.272095 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.272096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.272261 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.272328 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.272330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.273023 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.273025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.273649 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.273651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.274067 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.274068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.274148 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.274672 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.274677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.276071 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.277707 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.277708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.277709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.277902 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.278117 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.278118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.278118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.278182 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.278471 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.278472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.278521 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.278608 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.278717 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:35:26.279183 14439 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.279598 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.279449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42924)
0928 06:35:26.279526 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.279566 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:35:26.279937 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.279822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42932)
0928 06:35:26.279893 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.279918 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 06:35:26.280037 14439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38695: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
I20260928 06:35:26.311214 14439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } attempt: 1
I20260928 06:35:26.311754 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.311589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42944)
0928 06:35:26.311666 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.311719 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:35:26.312160 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.312053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42952)
0928 06:35:26.312102 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.312135 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:35:26.312278 14439 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38695: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
W20260928 06:35:26.315460 14521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:26.315670 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:26.315688 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:26.315758 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:26.316951 14521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:26.316990 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:26.317004 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:26.317018 14521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:26.317067 14521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:26.318666 14521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:37121,127.24.11.189:39433,127.24.11.188:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.11.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:26.318943 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:26.319186 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:26.324224 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.324180 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:26.324180 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:26.324555 14521 server_base.cc:1034] running on GCE node
I20260928 06:35:26.324805 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:26.325132 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:26.326298 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790577326326276 us; error 40 us; skew 500 ppm
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:26.328935 14521 init.cc:377] Logged in from keytab as kudu/127.24.11.188@KRBTEST.COM (short username kudu)
I20260928 06:35:26.329408 14521 webserver.cc:460] Webserver started at http://127.24.11.188:39157/ using document root <none> and password file <none>
I20260928 06:35:26.329713 14521 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:26.329787 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:26.329954 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:26.331049 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data/instance:
uuid: "33e7fbe81ef345a1b138213d58baa173"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "ff59b33dad1f8ecc9a2108e98a2f1bdd"
server_key_iv: "d41dedd3bef812498ff9ef8fe88014f2"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.331447 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal/instance:
uuid: "33e7fbe81ef345a1b138213d58baa173"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "ff59b33dad1f8ecc9a2108e98a2f1bdd"
server_key_iv: "d41dedd3bef812498ff9ef8fe88014f2"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.332861 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:35:26.333679 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:26.333887 14521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:35:26.333988 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "33e7fbe81ef345a1b138213d58baa173"
format_stamp: "Formatted at 2026-09-28 06:35:26 on dist-test-slave-38pg"
server_key: "ff59b33dad1f8ecc9a2108e98a2f1bdd"
server_key_iv: "d41dedd3bef812498ff9ef8fe88014f2"
server_key_version: "encryptionkey@0"
I20260928 06:35:26.334074 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:26.337874 14514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } attempt: 2
I20260928 06:35:26.338294 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.338153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42958)
0928 06:35:26.338215 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.338257 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:35:26.338622 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.338542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42974)
0928 06:35:26.338584 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.338606 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:35:26.338732 14514 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38695: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38695: connect: Connection refused (error 111)
I20260928 06:35:26.342507 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:26.344975 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:26.345141 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:26.349656 14521 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.188:38695
I20260928 06:35:26.349690 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.188:38695 every 8 connection(s)
I20260928 06:35:26.350051 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
I20260928 06:35:26.351095 14589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:26.352717 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.354815 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14521
I20260928 06:35:26.354902 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577055356400-24622-0/minicluster-data/master-2/wal/instance
I20260928 06:35:26.355101 24622 external_mini_cluster.cc:1428] Setting key d57399178735a4e6b00b22c3a00531f7
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:35:26.362860 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.353464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.24.11.190:37121)
0928 06:35:26.353563 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.353566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.353855 (+ 289us) server_negotiation.cc:407] Connection header received
0928 06:35:26.354380 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.354381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.354393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.354458 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.355251 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.355975 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.357274 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.357462 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.357542 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.359995 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.359997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.359997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.360007 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.361391 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.361862 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.361863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.361864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.361924 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.362187 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.362188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.362188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.362305 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.362378 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.362668 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.362751 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:26.363091 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.353460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:37121 (local address 127.0.0.1:50482)
0928 06:35:26.353730 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.353766 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.354268 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.354533 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.354537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.354821 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.355101 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.355106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.356066 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.356066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.357189 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.357191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.357613 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.357614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.357735 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.358304 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.358313 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.359898 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.361476 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.361483 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.361485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.361780 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.361993 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.361994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.361994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.362113 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.362383 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.362391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.362533 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.362635 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.362827 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":182,"threads_started":1}
I20260928 06:35:26.363761 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.364500 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.355521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42990 (local address 127.24.11.188:38695)
0928 06:35:26.355681 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.355685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.355706 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:35:26.355756 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.355758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.355773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.355838 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.356483 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.357462 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.358215 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.358531 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.358610 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.361178 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.361185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.361186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.361192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.362817 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.363484 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.363487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.363487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.363551 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.363770 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.363771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.363771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.363967 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.364113 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.364226 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.364382 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:26.370941 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.364113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.24.11.189:39433)
0928 06:35:26.364162 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.364164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.364172 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:26.364197 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.364199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.364209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.364262 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.364633 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.365101 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.365923 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.366215 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.366284 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.368122 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.368124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.368125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.368132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.369788 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.370140 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.370141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.370142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.370198 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.370423 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.370424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.370424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.370545 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.370619 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.370769 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.370851 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:35:26.370941 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.364042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39433 (local address 127.0.0.1:45514)
0928 06:35:26.364082 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.364091 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.364125 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.364279 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.364280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.364468 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.364529 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.364531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.365182 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.365184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.365841 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.365843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.366288 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.366289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.366350 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.366763 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.366769 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.368033 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.369884 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.369886 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.369886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.370052 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.370271 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.370272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.370272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.370332 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.370612 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.370613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.370657 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.370744 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.370843 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:35:26.371304 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:26.372733 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.365375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.24.11.190:37121)
0928 06:35:26.365415 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.365417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.365424 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:26.365442 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.365443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.365451 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.365512 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.365966 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.366545 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.367355 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.367525 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.367577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.369974 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.369975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.369976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.369983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.371667 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.371994 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.371995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.371995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.372048 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.372233 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.372235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.372235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.372346 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.372530 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.372565 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.372635 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:26.378418 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.371566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:42998)
0928 06:35:26.371614 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.371624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.371663 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.371788 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.371788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.371924 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.371951 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.371952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.372564 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.372566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.373276 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.373279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.373761 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.373762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.373811 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.374220 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.374224 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.375401 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.377168 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.377168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.377169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.377339 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.377648 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.377649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.377650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.377755 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.378042 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.378043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.378082 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.378175 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.378274 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:35:26.378418 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.371625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.24.11.188:38695)
0928 06:35:26.371659 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.371661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.371669 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:26.371721 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.371722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.371732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.371784 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.372022 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.372487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.373355 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.373678 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.373762 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.375484 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.375486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.375486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.375492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.377057 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.377458 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.377459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.377460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.377518 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.377836 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.377836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.377837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.377974 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.378051 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.378221 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.378319 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":19}
I20260928 06:35:26.380410 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: Bootstrap starting.
I20260928 06:35:26.381198 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.373343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.24.11.189:39433)
0928 06:35:26.373388 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.373390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.373431 (+ 41us) server_negotiation.cc:407] Connection header received
0928 06:35:26.373486 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.373487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.373496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.373558 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.373992 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.374522 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.375211 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.375538 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.375599 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.378260 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.378262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.378262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.378270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.379999 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.380375 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.380376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.380377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.380441 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.380695 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.380695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.380695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.380806 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.380932 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.381073 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.381135 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:26.381497 14589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:26.381887 14589 log.cc:824] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:26.382865 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: No bootstrap required, opened a new log
I20260928 06:35:26.383045 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:26.384876 14589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.385041 14589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e7fbe81ef345a1b138213d58baa173, State: Initialized, Role: FOLLOWER
I20260928 06:35:26.384188 14413 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50496
I20260928 06:35:26.385142 14589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.385440 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.385483 14589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:26.385578 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.387388 14439 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } attempt: 2
W20260928 06:35:26.387599 14606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:26.387964 14606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:26.387986 14521 master_runner.cc:186] Error getting master registration for 127.24.11.190:37121: OK, instance_id { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" instance_seqno: 1790577326043173 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:26.390949 14521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:35:26.395150 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.387872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.24.11.188:38695)
0928 06:35:26.387919 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.387921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.387928 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:26.387950 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.387951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.387961 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.388368 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.388400 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.389094 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.390119 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.390417 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.390747 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.392471 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.392473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.392473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.392480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.393949 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.394321 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.394322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.394322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.394381 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.394580 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.394581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.394581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.394728 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.394826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.394981 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.395056 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:35:26.395150 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.387741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:43010)
0928 06:35:26.387792 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.387804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.387844 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.388038 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.388039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.388236 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.388291 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.388294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.389208 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.389209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.390015 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.390018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.390511 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.390512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.390568 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.391067 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.391073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.392372 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.394049 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.394050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.394051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.394228 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.394448 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.394449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.394449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.394507 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.394820 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.394821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.394868 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.394956 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.395056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:35:26.396907 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: Bootstrap starting.
I20260928 06:35:26.397604 14439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:26.397900 14439 log.cc:824] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:26.398800 14439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: No bootstrap required, opened a new log
I20260928 06:35:26.399797 14439 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.399930 14439 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c0299cc0dc4b208cb8d34eb15e36aa, State: Initialized, Role: FOLLOWER
I20260928 06:35:26.400017 14439 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.400311 14439 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:26.400369 14413 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:35:26.400454 14413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:26.401222 14413 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.401449 14413 leader_election.cc:288] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [CANDIDATE]: Term 1 election: Requested vote from peers 76477bdbb9f24af596fa7b92cd2fe013 (127.24.11.189:39433), 33e7fbe81ef345a1b138213d58baa173 (127.24.11.188:38695)
I20260928 06:35:26.401854 14563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "33e7fbe81ef345a1b138213d58baa173"
I20260928 06:35:26.401990 14563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:26.400326 14609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.402904 14609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.402943 14563 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c0299cc0dc4b208cb8d34eb15e36aa in term 1.
I20260928 06:35:26.403407 14389 leader_election.cc:302] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e7fbe81ef345a1b138213d58baa173, e7c0299cc0dc4b208cb8d34eb15e36aa; no voters:
I20260928 06:35:26.403515 14609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:26.403779 14609 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [term 1 LEADER]: Becoming Leader. State: Replica: e7c0299cc0dc4b208cb8d34eb15e36aa, State: Running, Role: LEADER
I20260928 06:35:26.403870 14609 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.404285 14609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c0299cc0dc4b208cb8d34eb15e36aa. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.404356 14609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: This master's current role is: LEADER
W20260928 06:35:26.404933 14620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:26.404996 14620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:26.404004 14371 master_runner.cc:186] Error getting master registration for 127.24.11.189:39433: OK, instance_id { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" instance_seqno: 1790577326194720 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:26.406731 14371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:26.409950 14630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:26.410159 14630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:26.411150 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.402811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.24.11.190:37121)
0928 06:35:26.402849 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.402851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.403337 (+ 486us) server_negotiation.cc:407] Connection header received
0928 06:35:26.403477 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.403478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.403501 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.403533 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.404426 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.405132 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.406228 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.406537 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.406595 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.408278 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.408280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.408280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.408288 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.409924 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.410350 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.410350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.410351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.410421 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.410615 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.410616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.410616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.410729 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.410838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.410983 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.411068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:26.412539 14563 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [term 1 FOLLOWER]: Refusing update from remote peer e7c0299cc0dc4b208cb8d34eb15e36aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:26.412961 14609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:26.414398 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c0299cc0dc4b208cb8d34eb15e36aa. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.414481 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.416234 14609 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.416332 14609 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:26.416775 14595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.416872 14595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.416999 14632 mvcc.cc:205] Tried to move back new op lower bound from 7334204728982044672 to 7334204728951025664. Current Snapshot: MvccSnapshot[applied={T|T < 7334204728982044672}]
I20260928 06:35:26.417419 14633 mvcc.cc:205] Tried to move back new op lower bound from 7334204728982044672 to 7334204728951025664. Current Snapshot: MvccSnapshot[applied={T|T < 7334204728982044672}]
I20260928 06:35:26.418149 14630 catalog_manager.cc:1347] Generated new cluster ID: da0c1b6d7e784bf6a2c2512dc2893147
I20260928 06:35:26.418217 14630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:26.427212 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.403338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.24.11.188:38695)
0928 06:35:26.403633 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.403634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.411895 (+ 8261us) server_negotiation.cc:407] Connection header received
0928 06:35:26.411999 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.412000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.412011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.412043 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.412707 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.413366 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.414294 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.414568 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.414611 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.416192 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.416194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.416194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.416201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.417970 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.426071 (+ 8101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.426072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.426073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.426137 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.426374 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.426376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.426376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.426510 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.426616 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.427041 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.427124 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260928 06:35:26.429867 14514 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } attempt: 3
I20260928 06:35:26.432917 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.403191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.24.11.189:39433)
0928 06:35:26.403239 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.403240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.409726 (+ 6486us) server_negotiation.cc:407] Connection header received
0928 06:35:26.409840 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.409841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.409854 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.409916 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.411491 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.412253 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.414723 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.415029 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.415071 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.428898 (+ 13827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.428900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.428901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.428910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.431015 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.431658 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.431660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.431673 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.431734 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.432137 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.432151 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.432152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.432290 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.432374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.432712 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.432803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:35:26 dist-test-slave-38pg krb5kdc[14358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577326, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:35:26.438408 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.430163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38695 (local address 127.0.0.1:43024)
0928 06:35:26.430257 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:26.430267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:26.430305 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:26.431111 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:26.431112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:26.431247 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:26.431293 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.431295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.432236 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.432237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.433051 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:26.433053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.433576 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.433577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:26.433632 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.434057 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:26.434062 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:26.435581 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:26.437383 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.437384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.437384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.437611 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.437820 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:26.437821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:26.437821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:26.437875 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:26.438126 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:26.438127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:26.438168 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:35:26.438230 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:35:26.438315 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 06:35:26.438747 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.430271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.24.11.188:38695)
0928 06:35:26.430312 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.430313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.430325 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:26.430362 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.430363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.430373 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.430420 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.431389 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.432124 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.433140 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.433439 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.433497 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.435674 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:26.435675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:26.435676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:26.435682 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:26.437294 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.437688 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.437689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.437690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.437753 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:26.437936 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:26.437936 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:26.437936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:26.438048 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:26.438101 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.438573 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.438642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:26.440236 14514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: Bootstrap starting.
I20260928 06:35:26.440979 14514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:26.441278 14514 log.cc:824] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:26.449572 14630 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:26.450361 14630 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:26.453960 14514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: No bootstrap required, opened a new log
I20260928 06:35:26.455318 14630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa: Generated new TSK 0
I20260928 06:35:26.455714 14630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:35:26.456626 14514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.456825 14514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76477bdbb9f24af596fa7b92cd2fe013, State: Initialized, Role: FOLLOWER
I20260928 06:35:26.456964 14514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } }
I20260928 06:35:26.457340 14514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:26.457430 14487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:26.457643 14488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76477bdbb9f24af596fa7b92cd2fe013"
I20260928 06:35:26.458824 14487 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [term 1 FOLLOWER]: Refusing update from remote peer e7c0299cc0dc4b208cb8d34eb15e36aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:26.459164 14628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:26.460638 14488 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c0299cc0dc4b208cb8d34eb15e36aa in term 1.
I20260928 06:35:26.462309 14638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.462472 14638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.462576 14638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7c0299cc0dc4b208cb8d34eb15e36aa. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.462677 14638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.462771 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:26.463635 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:26.464967 14650 mvcc.cc:205] Tried to move back new op lower bound from 7334204729152126976 to 7334204728951025664. Current Snapshot: MvccSnapshot[applied={T|T < 7334204728982044672}]
I20260928 06:35:26.470185 14653 catalog_manager.cc:1259] Loaded cluster ID: da0c1b6d7e784bf6a2c2512dc2893147
I20260928 06:35:26.470253 14653 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: loading cluster ID for follower catalog manager: success
I20260928 06:35:26.471357 14653 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: acquiring CA information for follower catalog manager: success
I20260928 06:35:26.471674 14653 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76477bdbb9f24af596fa7b92cd2fe013: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:35:26.484242 14518 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.482499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.24.11.189:39433)
0928 06:35:26.482545 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.482547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.483499 (+ 952us) server_negotiation.cc:407] Connection header received
0928 06:35:26.483606 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.483607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.483618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.483650 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.484157 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45534: BlockingRecv error: recv got EOF from 127.0.0.1:45534 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:35:26.484998 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.482869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.24.11.188:38695)
0928 06:35:26.482908 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.482910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.484498 (+ 1588us) server_negotiation.cc:407] Connection header received
0928 06:35:26.484595 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.484596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.484605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.484636 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.484935 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43034: BlockingRecv error: recv got EOF from 127.0.0.1:43034 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:35:26.485839 14441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.483221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.24.11.190:37121)
0928 06:35:26.483267 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.483269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.485226 (+ 1957us) server_negotiation.cc:407] Connection header received
0928 06:35:26.485396 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.485397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.485407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.485437 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.485780 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50512: BlockingRecv error: recv got EOF from 127.0.0.1:50512 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:35:26.497017 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.490214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.24.11.189:39433)
0928 06:35:26.490249 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.490251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.493698 (+ 3447us) server_negotiation.cc:407] Connection header received
0928 06:35:26.493810 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.493811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.493853 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.493895 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.494094 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.494908 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.496068 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.496373 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.496414 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.496634 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.496757 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.496805 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.496918 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.496936 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:26.500452 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.499497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.24.11.188:38695)
0928 06:35:26.499537 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.499539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.499765 (+ 226us) server_negotiation.cc:407] Connection header received
0928 06:35:26.499870 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.499872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.499910 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.499941 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.500399 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43050: BlockingRecv error: recv got EOF from 127.0.0.1:43050 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:26.505904 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.502687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.24.11.189:39433)
0928 06:35:26.502722 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.502723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.502943 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:35:26.503039 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.503040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.503048 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.503075 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.503236 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.503972 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.505083 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.505390 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.505432 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.505651 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.505683 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.505711 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.505815 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.505833 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:35:26.508958 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.508136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.24.11.188:38695)
0928 06:35:26.508172 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.508173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.508420 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:35:26.508540 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.508541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.508554 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.508589 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.508907 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43052: BlockingRecv error: recv got EOF from 127.0.0.1:43052 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:26.515219 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.512008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.24.11.189:39433)
0928 06:35:26.512044 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.512045 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.512272 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:35:26.512378 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.512379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.512387 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.512415 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.512575 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.513312 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.514413 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.514731 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.514770 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.514961 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.514994 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.515021 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.515123 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.515146 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:26.518193 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.517403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.24.11.188:38695)
0928 06:35:26.517443 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.517445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.517662 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:35:26.517759 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.517760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.517772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.517802 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.518140 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43068: BlockingRecv error: recv got EOF from 127.0.0.1:43068 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:26.526513 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.523261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.24.11.189:39433)
0928 06:35:26.523310 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.523312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.523568 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:35:26.523663 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.523664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.523672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.523699 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.523859 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.524605 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.525729 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.526023 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.526062 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.526244 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.526276 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.526314 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.526418 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.526435 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:35:26.529482 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.528698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.24.11.188:38695)
0928 06:35:26.528735 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.528736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.528934 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:35:26.529029 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.529030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.529041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.529072 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.529425 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43076: BlockingRecv error: recv got EOF from 127.0.0.1:43076 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:35:26.542026 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.538717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.24.11.189:39433)
0928 06:35:26.538759 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.538761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.539028 (+ 267us) server_negotiation.cc:407] Connection header received
0928 06:35:26.539132 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.539133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.539143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.539172 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.539358 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.540102 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.541172 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.541483 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.541522 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.541751 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.541786 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.541814 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.541921 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.541940 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:35:26.545178 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.544374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.24.11.188:38695)
0928 06:35:26.544415 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.544417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.544629 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:35:26.544728 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.544729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.544741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.544771 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.545121 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43088: BlockingRecv error: recv got EOF from 127.0.0.1:43088 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:35:26.566005 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.562557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.24.11.189:39433)
0928 06:35:26.562601 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.562603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.562885 (+ 282us) server_negotiation.cc:407] Connection header received
0928 06:35:26.563017 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.563018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.563030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.563060 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.563262 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.564001 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.565126 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.565434 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.565481 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.565730 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.565765 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.565794 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.565904 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.565924 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:35:26.569265 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.568385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.24.11.188:38695)
0928 06:35:26.568425 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.568438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.568690 (+ 252us) server_negotiation.cc:407] Connection header received
0928 06:35:26.568793 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.568794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.568806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.568838 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.569205 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43102: BlockingRecv error: recv got EOF from 127.0.0.1:43102 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:26.605981 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.602759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.24.11.189:39433)
0928 06:35:26.602807 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.602809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.602981 (+ 172us) server_negotiation.cc:407] Connection header received
0928 06:35:26.603022 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.603023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.603036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.603106 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.603253 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.603992 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.605124 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.605423 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.605501 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.605701 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.605739 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.605797 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.605874 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.605895 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:35:26.608757 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.607887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.24.11.188:38695)
0928 06:35:26.607943 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.607945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.608255 (+ 310us) server_negotiation.cc:407] Connection header received
0928 06:35:26.608295 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.608296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.608308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.608367 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.608698 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43104: BlockingRecv error: recv got EOF from 127.0.0.1:43104 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:26.676283 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.674077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.24.11.189:39433)
0928 06:35:26.674154 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.674156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.674252 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:35:26.674293 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.674294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.674304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.674377 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.674495 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.674973 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.675707 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.675896 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.675949 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.676075 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.676102 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.676140 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.676210 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.676225 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:35:26.678675 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.678082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.24.11.188:38695)
0928 06:35:26.678129 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.678132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.678177 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:35:26.678242 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.678243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.678254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.678310 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.678615 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43118: BlockingRecv error: recv got EOF from 127.0.0.1:43118 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:26.810329 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:26.808053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.24.11.189:39433)
0928 06:35:26.808110 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.808112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.808186 (+ 74us) server_negotiation.cc:407] Connection header received
0928 06:35:26.808228 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.808229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.808238 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.808289 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:26.808410 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.808895 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.809688 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:26.809881 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:26.809948 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:26.810073 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:26.810103 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:26.810142 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:26.810193 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:35:26.810208 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:35:26.812718 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:26.812118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.24.11.188:38695)
0928 06:35:26.812195 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:26.812197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:26.812242 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:35:26.812304 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:26.812306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:26.812316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:26.812360 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:26.812637 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43132: BlockingRecv error: recv got EOF from 127.0.0.1:43132 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:35:27.072175 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.069955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.24.11.189:39433)
0928 06:35:27.070035 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.070037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.070047 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:27.070078 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.070079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.070088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.070138 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.070297 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.070855 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.071600 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.071785 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.071845 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.071978 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.072005 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.072044 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.072098 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.072114 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:35:27.074499 14593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:27.073902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.24.11.188:38695)
0928 06:35:27.073966 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.073968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.073978 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:27.074050 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.074051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.074063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.074114 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:27.074437 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43138: BlockingRecv error: recv got EOF from 127.0.0.1:43138 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:35:27.079885 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c0299cc0dc4b208cb8d34eb15e36aa" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 37121 } } peers { permanent_uuid: "76477bdbb9f24af596fa7b92cd2fe013" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39433 } } peers { permanent_uuid: "33e7fbe81ef345a1b138213d58baa173" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38695 } } }
I20260928 06:35:27.080032 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7c0299cc0dc4b208cb8d34eb15e36aa [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:27.388490 14606 catalog_manager.cc:1259] Loaded cluster ID: da0c1b6d7e784bf6a2c2512dc2893147
I20260928 06:35:27.388567 14606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: loading cluster ID for follower catalog manager: success
I20260928 06:35:27.389452 14606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: acquiring CA information for follower catalog manager: success
I20260928 06:35:27.389791 14606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33e7fbe81ef345a1b138213d58baa173: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:27.590142 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.587610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.24.11.189:39433)
0928 06:35:27.587734 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.587736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.587794 (+ 58us) server_negotiation.cc:407] Connection header received
0928 06:35:27.587883 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.587884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.587902 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.587975 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.588107 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.588598 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.589385 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.589637 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.589720 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.589867 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.589909 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.589966 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.590023 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.590047 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260928 06:35:27.593945 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.591672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.24.11.188:38695)
0928 06:35:27.591790 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.591792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.591801 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:27.591828 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.591829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.591886 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.591959 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.592063 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.592529 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.593187 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.593373 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.593421 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.593583 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.593693 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.593752 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.593801 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.593817 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260928 06:35:27.598282 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.595714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.24.11.188:38695)
0928 06:35:27.595779 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.595781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.595903 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:35:27.595945 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.595946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.595956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.595999 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.596155 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.596875 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.597643 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.597898 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.597966 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.598062 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.598092 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.598147 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.598181 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.598203 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:35:27.598701 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.595549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.24.11.189:39433)
0928 06:35:27.595590 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.595592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.595681 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:35:27.595777 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.595778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.595789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.595819 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.595976 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.596716 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.597846 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.598161 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.598226 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.598462 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.598492 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.598518 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.598603 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.598624 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:27.599215 14788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:27.595643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.24.11.190:37121)
0928 06:35:27.595742 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:27.595744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:27.595752 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:35:27.595802 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:27.595803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:27.595848 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:27.596125 (+ 277us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:27.596166 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.596642 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.598265 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:27.598558 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:27.598648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:27.598868 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:27.598973 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:27.599037 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:27.599113 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:35:27.599138 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260928 06:35:27.600935 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14371
I20260928 06:35:27.606184 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14446
I20260928 06:35:27.611094 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14521
2026-09-28T06:35:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2845 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:35:27 dist-test-slave-38pg krb5kdc[14792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:35:27 dist-test-slave-38pg krb5kdc[14792](info): set up 2 sockets
Sep 28 06:35:27 dist-test-slave-38pg krb5kdc[14792](info): commencing operation
krb5kdc: starting...
W20260928 06:35:28.667182 24622 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/test-user.keytab.
Sep 28 06:35:28 dist-test-slave-38pg krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.11.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
Entry for principal kudu/127.24.11.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab.
I20260928 06:35:28.790575 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39903
--webserver_interface=127.24.11.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:35:28.873499 14805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:28.873767 14805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:28.873795 14805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:28.873816 14805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:28.875273 14805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:28.875329 14805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:28.875344 14805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:28.875356 14805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:28.875370 14805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:28.876955 14805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.190.keytab
--principal=kudu/127.24.11.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.190:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.11.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:28.877158 14805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:28.877347 14805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:28.882223 14813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:28.882230 14811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.882320 14805 server_base.cc:1034] running on GCE node
W20260928 06:35:28.882234 14810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:28.882658 14805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:28.883000 14805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:28.884166 14805 hybrid_clock.cc:648] HybridClock initialized: now 1790577328884157 us; error 29 us; skew 500 ppm
Sep 28 06:35:28 dist-test-slave-38pg krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:28.886664 14805 init.cc:377] Logged in from keytab as kudu/127.24.11.190@KRBTEST.COM (short username kudu)
I20260928 06:35:28.887099 14805 webserver.cc:460] Webserver started at http://127.24.11.190:33591/ using document root <none> and password file <none>
I20260928 06:35:28.887326 14805 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:28.887398 14805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:28.887550 14805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:28.888639 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data/instance:
uuid: "933e4d14c8c74898a7775f0e18335557"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-38pg"
server_key: "5aa95efa9043376f7433b181b5d69467"
server_key_iv: "ab5f190ecd5a99b11fcdf9bc6f42fc54"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.889010 14805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance:
uuid: "933e4d14c8c74898a7775f0e18335557"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-38pg"
server_key: "5aa95efa9043376f7433b181b5d69467"
server_key_iv: "ab5f190ecd5a99b11fcdf9bc6f42fc54"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.890470 14805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:35:28.891196 14820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.891364 14805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:28.891467 14805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal
uuid: "933e4d14c8c74898a7775f0e18335557"
format_stamp: "Formatted at 2026-09-28 06:35:28 on dist-test-slave-38pg"
server_key: "5aa95efa9043376f7433b181b5d69467"
server_key_iv: "ab5f190ecd5a99b11fcdf9bc6f42fc54"
server_key_version: "encryptionkey@0"
I20260928 06:35:28.891541 14805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:28.901379 14805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:28.903239 14805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:28.903416 14805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:28.907599 14805 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.190:39903
I20260928 06:35:28.907634 14872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.190:39903 every 8 connection(s)
I20260928 06:35:28.908003 14805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/data/info.pb
I20260928 06:35:28.908981 14873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:28.910652 14873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:28.916128 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14805
I20260928 06:35:28.916225 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-0/wal/instance
I20260928 06:35:28.916453 24622 external_mini_cluster.cc:1428] Setting key 708374d0ba691d455e199bab9ffcbe4d
Sep 28 06:35:28 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:28.920262 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.911378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.24.11.190:39903)
0928 06:35:28.911555 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:28.911560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:28.911635 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:35:28.912124 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:28.912125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:28.912142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:28.912220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:28.912734 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.913599 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.914440 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.914751 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.914835 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.916914 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:28.916922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:28.916922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:28.916929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:28.918601 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.919036 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.919039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.919039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.919145 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:28.919378 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:28.919379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:28.919380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:28.919678 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:28.919764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:28.919965 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 06:35:28.920115 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260928 06:35:28.920266 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.911323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39903 (local address 127.0.0.1:37154)
0928 06:35:28.911503 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.911547 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:28.911996 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:28.912244 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:28.912247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:28.912497 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:28.912653 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.912656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.913696 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.913697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.914367 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:28.914369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:28.914855 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:28.914855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:28.914909 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:28.915393 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:28.915401 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:28.916831 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:28.918683 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.918689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.918692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.918955 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.919213 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:28.919214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:28.919214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:28.919305 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:28.919746 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:28.919751 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:28.919825 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 06:35:28.919898 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:35:28.920025 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260928 06:35:28.921371 14873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:28.921790 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.921633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44612)
0928 06:35:28.921695 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.921752 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 06:35:28.921976 14821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
I20260928 06:35:28.922495 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.922377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44620)
0928 06:35:28.922430 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.922465 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:35:28.922637 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39139: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.11.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
Entry for principal kudu/127.24.11.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab.
I20260928 06:35:28.942917 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:39139
--webserver_interface=127.24.11.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:28.971890 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } attempt: 1
I20260928 06:35:28.972438 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.972297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44632)
0928 06:35:28.972365 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.972407 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 06:35:28.972811 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:28.972698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44642)
0928 06:35:28.972771 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:28.972793 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 06:35:28.972898 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39139: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
W20260928 06:35:29.026098 14880 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:29.026362 14880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:29.026396 14880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:29.026433 14880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:29.027572 14880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:29.027642 14880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:29.027665 14880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:29.027688 14880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:29.027717 14880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:29.029219 14880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.189.keytab
--principal=kudu/127.24.11.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.189:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.11.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:29.029503 14880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:29.029749 14880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:29.034821 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:29.034859 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:29.034988 14880 server_base.cc:1034] running on GCE node
W20260928 06:35:29.034832 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:29.035321 14880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:29.035615 14880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:29.036762 14880 hybrid_clock.cc:648] HybridClock initialized: now 1790577329036743 us; error 41 us; skew 500 ppm
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:29.039233 14880 init.cc:377] Logged in from keytab as kudu/127.24.11.189@KRBTEST.COM (short username kudu)
I20260928 06:35:29.039687 14880 webserver.cc:460] Webserver started at http://127.24.11.189:35507/ using document root <none> and password file <none>
I20260928 06:35:29.039871 14880 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:29.039914 14880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:29.040004 14880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:29.041029 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } attempt: 2
I20260928 06:35:29.041018 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data/instance:
uuid: "a12a52cb456d42d0aaceefb34e420d85"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "fdabe531322b0488fb067e4124c327bd"
server_key_iv: "eca63e631eeb52639496fe0be95ae501"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.041385 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal/instance:
uuid: "a12a52cb456d42d0aaceefb34e420d85"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "fdabe531322b0488fb067e4124c327bd"
server_key_iv: "eca63e631eeb52639496fe0be95ae501"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.041410 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.041282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44654)
0928 06:35:29.041346 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.041383 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:35:29.041785 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.041697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44660)
0928 06:35:29.041738 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.041761 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:35:29.041918 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.189:39139: Network error: Client connection negotiation failed: client connection to 127.24.11.189:39139: connect: Connection refused (error 111)
I20260928 06:35:29.042838 14880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:35:29.043596 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:29.043766 14880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:35:29.043872 14880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal
uuid: "a12a52cb456d42d0aaceefb34e420d85"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "fdabe531322b0488fb067e4124c327bd"
server_key_iv: "eca63e631eeb52639496fe0be95ae501"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.043949 14880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:29.090221 14880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:29.092159 14880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:29.092334 14880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:29.096668 14880 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.189:39139
I20260928 06:35:29.096722 14947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.189:39139 every 8 connection(s)
I20260928 06:35:29.097108 14880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/data/info.pb
I20260928 06:35:29.098120 14948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:29.098207 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14880
I20260928 06:35:29.098333 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-1/wal/instance
I20260928 06:35:29.098526 24622 external_mini_cluster.cc:1428] Setting key d781cf1b18012ea2d12c546b0ee90d97
I20260928 06:35:29.099808 14948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.24.11.188@KRBTEST.COM; defaulting to no policy
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.11.188@KRBTEST.COM" created.
I20260928 06:35:29.109606 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.100579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37168 (local address 127.24.11.190:39903)
0928 06:35:29.100649 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.100652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.100836 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:35:29.101304 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.101305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.101322 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.101399 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.102051 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.102879 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.103866 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.104205 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.104292 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.106703 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.106705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.106705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.106712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.108056 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.108440 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.108440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.108441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.108491 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.108827 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.108828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.108829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.108953 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.109010 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.109330 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.109421 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:35:29.109643 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.100509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39903 (local address 127.0.0.1:37168)
0928 06:35:29.100703 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.100745 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.101211 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.101470 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.101474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.101759 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.101956 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.101960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.102987 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.102987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.103780 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.103782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.104314 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.104315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.104432 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.105023 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.105031 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.106613 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.108137 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.108145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.108147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.108375 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.108575 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.108576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.108576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.108749 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.109026 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.109032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.109192 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.109283 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.109418 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260928 06:35:29.111325 14948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:29.118870 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.111608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44674)
0928 06:35:29.111823 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.111835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.111886 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.112026 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.112027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.112167 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.112213 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.112214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.113076 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.113077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.113668 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.113670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.114173 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.114174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.114237 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.114692 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.114697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.116134 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.117890 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.117891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.117891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.118046 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.118233 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.118234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.118234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.118296 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.118591 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.118592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.118640 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.118710 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.118791 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:35:29.118938 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.111683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.24.11.189:39139)
0928 06:35:29.111844 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.111849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.111886 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:35:29.111942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.111945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.111957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.112017 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.112287 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.112967 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.113755 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.114086 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.114174 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.116224 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.116231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.116232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.116238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.117814 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.118117 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.118119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.118120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.118173 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.118376 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.118376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.118377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.118519 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.118583 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.118751 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.118840 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260928 06:35:29.119578 14948 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:29.119957 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.119838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:44968)
0928 06:35:29.119888 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.119931 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:35:29.120085 14897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
I20260928 06:35:29.120476 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.120374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:44980)
0928 06:35:29.120425 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.120458 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:35:29.120595 14948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
Entry for principal kudu/127.24.11.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab.
I20260928 06:35:29.124608 24622 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwumdS4/build/release/bin/kudu
/tmp/dist-test-taskwumdS4/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:38487
--webserver_interface=127.24.11.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.11.148:36629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:35:29.133087 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } attempt: 3
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:29.140241 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.133618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.24.11.189:39139)
0928 06:35:29.133685 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.133687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.133698 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:35:29.133804 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.133805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.133821 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.133889 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.134205 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.134719 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.135424 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.135651 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.135726 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.137687 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.137689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.137689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.137699 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.139068 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.139437 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.139438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.139438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.139495 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.139700 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.139701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.139702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.139820 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.139897 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.140053 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.140135 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:35:29.140233 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.133518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44688)
0928 06:35:29.133614 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.133625 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.133694 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.133894 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.133895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.134054 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.134118 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.134119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.134805 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.134806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.135332 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.135334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.135743 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.135744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.135821 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.136307 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.136311 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.137568 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.139166 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.139168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.139168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.139348 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.139564 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.139565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.139566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.139633 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.139888 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.139890 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.139937 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.140027 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.140135 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 06:35:29.140771 14873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:35:29.141124 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.141016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:44986)
0928 06:35:29.141065 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.141094 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:35:29.141462 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.141406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:44992)
0928 06:35:29.141429 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.141449 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 06:35:29.141572 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
I20260928 06:35:29.169824 14948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 1
I20260928 06:35:29.170397 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.170247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:44996)
0928 06:35:29.170319 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.170366 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 06:35:29.170820 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.170720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45002)
0928 06:35:29.170769 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.170798 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 06:35:29.170941 14948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
I20260928 06:35:29.172736 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 1
I20260928 06:35:29.173120 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.172991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45012)
0928 06:35:29.173056 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.173094 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 06:35:29.173466 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.173360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45026)
0928 06:35:29.173410 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.173436 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:35:29.173588 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
W20260928 06:35:29.207238 14955 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:35:29.207484 14955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:35:29.207515 14955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:35:29.207574 14955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:35:29.208714 14955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:35:29.208779 14955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:35:29.208802 14955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:35:29.208825 14955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:35:29.208846 14955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:35:29.210327 14955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.11.148:36629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.11.190:39903,127.24.11.189:39139,127.24.11.188:38487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/krb5kdc/kudu_127.24.11.188.keytab
--principal=kudu/127.24.11.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.11.188:38487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.11.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:35:29.210608 14955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:35:29.210839 14955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:35:29.215656 14960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:35:29.215667 14963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:29.215904 14955 server_base.cc:1034] running on GCE node
W20260928 06:35:29.215694 14961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:35:29.216179 14955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:35:29.216471 14955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:35:29.217594 14955 hybrid_clock.cc:648] HybridClock initialized: now 1790577329217578 us; error 28 us; skew 500 ppm
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:35:29.220172 14955 init.cc:377] Logged in from keytab as kudu/127.24.11.188@KRBTEST.COM (short username kudu)
I20260928 06:35:29.220659 14955 webserver.cc:460] Webserver started at http://127.24.11.188:40153/ using document root <none> and password file <none>
I20260928 06:35:29.220880 14955 fs_manager.cc:359] Metadata directory not provided
I20260928 06:35:29.220925 14955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:35:29.221071 14955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:35:29.222189 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data/instance:
uuid: "1f0fbabaabea4f9f8ba203c8838ff187"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "92a0dc4337f8daa732ab23f50746060a"
server_key_iv: "fd535bdf19362e6afdc1caf19a99a576"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.222558 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal/instance:
uuid: "1f0fbabaabea4f9f8ba203c8838ff187"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "92a0dc4337f8daa732ab23f50746060a"
server_key_iv: "fd535bdf19362e6afdc1caf19a99a576"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.223863 14955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:35:29.224565 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:35:29.224748 14955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:35:29.224835 14955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data,/tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal
uuid: "1f0fbabaabea4f9f8ba203c8838ff187"
format_stamp: "Formatted at 2026-09-28 06:35:29 on dist-test-slave-38pg"
server_key: "92a0dc4337f8daa732ab23f50746060a"
server_key_iv: "fd535bdf19362e6afdc1caf19a99a576"
server_key_version: "encryptionkey@0"
I20260928 06:35:29.224912 14955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:35:29.239110 14948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 2
I20260928 06:35:29.239580 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.239445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45038)
0928 06:35:29.239511 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.239550 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:35:29.239950 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.239848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45052)
0928 06:35:29.239898 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.239924 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:35:29.240036 14948 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
I20260928 06:35:29.248732 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 2
I20260928 06:35:29.249202 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.249070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45066)
0928 06:35:29.249133 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.249172 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 06:35:29.249852 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.249725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45076)
0928 06:35:29.249781 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.249817 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 06:35:29.249972 14873 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.11.188:38487: Network error: Client connection negotiation failed: client connection to 127.24.11.188:38487: connect: Connection refused (error 111)
I20260928 06:35:29.263154 14955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:35:29.265617 14955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:35:29.265740 14955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:35:29.269479 14955 rpc_server.cc:307] RPC server started. Bound to: 127.24.11.188:38487
I20260928 06:35:29.269560 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.11.188:38487 every 8 connection(s)
I20260928 06:35:29.269901 14955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/data/info.pb
I20260928 06:35:29.270866 15023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:35:29.272365 15023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:29.279896 24622 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwumdS4/build/release/bin/kudu as pid 14955
I20260928 06:35:29.280021 24622 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwumdS4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577055356400-24622-0/minicluster-data/master-2/wal/instance
I20260928 06:35:29.280254 24622 external_mini_cluster.cc:1428] Setting key b88af6691dd2f08d188109df2d6c2c20
I20260928 06:35:29.281489 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.273052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.24.11.190:39903)
0928 06:35:29.273106 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.273108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.273298 (+ 190us) server_negotiation.cc:407] Connection header received
0928 06:35:29.273759 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.273759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.273768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.273836 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.274462 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.274936 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.275943 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.276128 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.276184 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.278333 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.278335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.278335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.278343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.279769 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.280263 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.280264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.280264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.280327 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.280608 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.280609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.280609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.280768 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.280839 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.281269 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.281370 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:29.281502 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.272995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.190:39903 (local address 127.0.0.1:37182)
0928 06:35:29.273162 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.273204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.273680 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.273929 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.273934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.274185 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.274388 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.274392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.275050 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.275051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.275862 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.275864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.276214 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.276215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.276325 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.276809 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.276817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.278230 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.279856 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.279865 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.279867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.280185 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.280416 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.280417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.280417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.280535 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.280875 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.280884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.281064 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.281196 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.281334 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260928 06:35:29.282138 15023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
I20260928 06:35:29.289952 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.282502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.24.11.189:39139)
0928 06:35:29.282547 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.282549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.282595 (+ 46us) server_negotiation.cc:407] Connection header received
0928 06:35:29.282665 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.282666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.282674 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.282716 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.283006 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.283737 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.284643 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.284926 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.284995 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.287159 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.287160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.287161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.287169 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.288741 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.289101 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.289102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.289102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.289165 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.289386 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.289387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.289387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.289517 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.289606 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.289752 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.289846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:29.290037 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.280705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.24.11.188:38487)
0928 06:35:29.280921 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.280926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.280964 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:35:29.280984 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.280986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.281000 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.281070 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.281873 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.282926 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.283949 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.284387 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.284460 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.286775 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.286784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.286784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.286790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.288677 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.289110 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.289113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.289114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.289167 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.289386 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.289386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.289387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.289585 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.289734 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.289793 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.289916 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":185,"thread_start_us":123,"threads_started":1}
I20260928 06:35:29.289990 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.282428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.189:39139 (local address 127.0.0.1:44694)
0928 06:35:29.282548 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.282558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.282608 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.282719 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.282720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.282868 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.282917 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.282919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.283815 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.283816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.284555 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.284557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.285002 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.285002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.285054 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.285438 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.285443 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.287073 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.288821 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.288822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.288822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.289023 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.289241 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.289242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.289242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.289312 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.289593 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.289593 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.289641 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.289711 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.289857 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 06:35:29.290550 15023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.188@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.190@KRBTEST.COM
I20260928 06:35:29.297796 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.290892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.24.11.188:38487)
0928 06:35:29.290927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.290929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.290935 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:35:29.290954 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.290955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.290962 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.291016 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.291293 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.291922 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.292645 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.292962 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.293026 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.295032 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.295034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.295034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.295040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.296618 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.296984 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.296984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.296985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.297032 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.297226 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.297227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.297227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.297331 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.297379 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.297587 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.297689 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":17}
I20260928 06:35:29.297797 15025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.290783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45102)
0928 06:35:29.290836 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.290847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.290891 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.291021 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.291022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.291172 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.291205 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.291207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.291993 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.291993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.292574 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.292575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.293034 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.293035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.293084 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.293486 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.293490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.294968 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.296698 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.296699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.296700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.296908 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.297098 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.297099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.297099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.297162 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.297394 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.297395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.297440 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.297523 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.297648 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:35:29.298961 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.290876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.24.11.190:39903)
0928 06:35:29.290920 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.290922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.291020 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:35:29.291068 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.291069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.291079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.291134 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.291667 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.292235 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.293033 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.293295 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.293370 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.295887 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.295888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.295888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.295896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.297472 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.298122 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.298123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.298123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.298184 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.298431 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.298431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.298432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.298571 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.298744 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.298781 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.298863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:35:29.299400 15023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: Bootstrap starting.
I20260928 06:35:29.300381 15023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:29.300724 15023 log.cc:824] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:29.301491 15023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: No bootstrap required, opened a new log
I20260928 06:35:29.303136 15023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.303258 15023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0fbabaabea4f9f8ba203c8838ff187, State: Initialized, Role: FOLLOWER
I20260928 06:35:29.303339 15023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.303659 15023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:29.303629 15029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.303714 15029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.11.189@KRBTEST.COM
W20260928 06:35:29.305547 15040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:29.305599 15040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:29.307410 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.299595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.24.11.189:39139)
0928 06:35:29.299640 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.299641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.299681 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:35:29.299734 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.299735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.299748 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.299817 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.300227 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.300939 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.301756 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.302037 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.302109 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.304442 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.304443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.304443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.304449 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.306208 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.306575 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.306576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.306577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.306640 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.306855 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.306856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.306856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.306989 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.307164 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.307205 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.307295 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:29.309006 24622 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:35:29.306195 14955 master_runner.cc:186] Error getting master registration for 127.24.11.190:39903: OK, instance_id { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" instance_seqno: 1790577328906399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:29.309345 14955 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:29.309986 14847 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "933e4d14c8c74898a7775f0e18335557"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37198
I20260928 06:35:29.331166 14948 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 3
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577329, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.189@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:35:29.339601 14951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.331556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45116)
0928 06:35:29.331631 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.331646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.331711 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.331894 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.331895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.332099 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.332155 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.332157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.333067 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.333069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.333920 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.333924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.334417 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.334418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.334479 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.335018 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.335023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.336526 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.338427 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.338428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.338429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.338606 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.338842 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.338844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.338844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.338919 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.339254 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.339255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.339306 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.339390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.339493 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:35:29.339607 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.331628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.24.11.188:38487)
0928 06:35:29.331700 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.331702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.331714 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:35:29.331795 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.331796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.331807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.331873 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.332242 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.332977 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.334014 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.334339 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.334405 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.336631 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.336633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.336634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.336641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.338343 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.338697 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.338698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.338699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.338757 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.338990 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.338991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.338991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.339180 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.339246 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.339424 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.339507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:29.341717 14948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: Bootstrap starting.
I20260928 06:35:29.342367 14948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:29.342661 14948 log.cc:824] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:29.343390 14948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: No bootstrap required, opened a new log
I20260928 06:35:29.344811 14948 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.344923 14948 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12a52cb456d42d0aaceefb34e420d85, State: Initialized, Role: FOLLOWER
I20260928 06:35:29.345008 14948 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.345305 15043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.345373 14948 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:29.345398 15043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:35:29.347244 15054 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:29.347339 15054 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:29.349097 14873 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } attempt: 3
I20260928 06:35:29.347437 14880 master_runner.cc:186] Error getting master registration for 127.24.11.190:39903: OK, instance_id { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" instance_seqno: 1790577328906399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:35:29.350507 14880 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:35:29 dist-test-slave-38pg krb5kdc[14792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577328, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.11.190@KRBTEST.COM for kudu/127.24.11.188@KRBTEST.COM
I20260928 06:35:29.357267 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.349372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.11.188:38487 (local address 127.0.0.1:45128)
0928 06:35:29.349482 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:35:29.349493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:35:29.349681 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:35:29.349876 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:35:29.349877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:35:29.350060 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:35:29.350105 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.350107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.350913 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.350914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.351718 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:35:29.351720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.352169 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.352170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:35:29.352224 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.352713 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:35:29.352718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:35:29.354149 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:35:29.356188 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.356189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.356190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.356403 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.356608 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:35:29.356609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:35:29.356609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:35:29.356669 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:35:29.356938 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:35:29.356939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:35:29.356982 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:35:29.357062 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:35:29.357169 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 06:35:29.357267 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.349605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.24.11.188:38487)
0928 06:35:29.349646 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.349648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.349657 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:35:29.349765 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.349765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.349775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.349830 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.350186 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.350813 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.351801 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.352113 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.352172 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.354239 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.354240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.354241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.354248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.356095 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.356466 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.356468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.356468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.356521 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.356719 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.356720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.356721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.356866 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.356923 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.357101 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.357172 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:35:29.358899 14873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: Bootstrap starting.
I20260928 06:35:29.359639 14873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: Neither blocks nor log segments found. Creating new log.
I20260928 06:35:29.359938 14873 log.cc:824] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: Log is configured to *not* fsync() on all Append() calls
I20260928 06:35:29.360780 14873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: No bootstrap required, opened a new log
I20260928 06:35:29.362198 14873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.362339 14873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 933e4d14c8c74898a7775f0e18335557, State: Initialized, Role: FOLLOWER
I20260928 06:35:29.362411 14873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.362691 15057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.362732 14847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:35:29.362826 15057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:29.362847 14847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:29.363430 14847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.363648 14847 leader_election.cc:288] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [CANDIDATE]: Term 1 election: Requested vote from peers a12a52cb456d42d0aaceefb34e420d85 (127.24.11.189:39139), 1f0fbabaabea4f9f8ba203c8838ff187 (127.24.11.188:38487)
I20260928 06:35:29.363991 14922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "933e4d14c8c74898a7775f0e18335557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a12a52cb456d42d0aaceefb34e420d85"
I20260928 06:35:29.364143 14922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:35:29.362710 14873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:35:29.365108 14922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933e4d14c8c74898a7775f0e18335557 in term 1.
I20260928 06:35:29.365393 14821 leader_election.cc:302] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 933e4d14c8c74898a7775f0e18335557, a12a52cb456d42d0aaceefb34e420d85; no voters:
I20260928 06:35:29.365757 14997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "933e4d14c8c74898a7775f0e18335557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f0fbabaabea4f9f8ba203c8838ff187"
I20260928 06:35:29.365904 14997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [term 0 FOLLOWER]: Advancing to term 1
W20260928 06:35:29.366825 15072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:35:29.366886 15072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:35:29.366887 14997 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 933e4d14c8c74898a7775f0e18335557 in term 1.
I20260928 06:35:29.367148 15057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:35:29.367278 15057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [term 1 LEADER]: Becoming Leader. State: Replica: 933e4d14c8c74898a7775f0e18335557, State: Running, Role: LEADER
I20260928 06:35:29.367373 15057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } }
I20260928 06:35:29.367733 15057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933e4d14c8c74898a7775f0e18335557. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.367803 15057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:29.368392 14805 master_runner.cc:263] No leader master found from master 933e4d14c8c74898a7775f0e18335557
I20260928 06:35:29.368444 14805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:35:29.374771 15076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:35:29.375052 15076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:35:29.386642 14997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [term 1 FOLLOWER]: Refusing update from remote peer 933e4d14c8c74898a7775f0e18335557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:29.387009 14922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [term 1 FOLLOWER]: Refusing update from remote peer 933e4d14c8c74898a7775f0e18335557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:35:29.387326 15057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:29.387817 15057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:35:29.388741 15029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933e4d14c8c74898a7775f0e18335557. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.388832 15029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:29.389389 15043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 933e4d14c8c74898a7775f0e18335557. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.389472 15043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:29.391444 14953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.368097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44714 (local address 127.24.11.189:39139)
0928 06:35:29.368140 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.368142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.369124 (+ 982us) server_negotiation.cc:407] Connection header received
0928 06:35:29.370928 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.370929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.370939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.370968 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.373480 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.374202 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.378357 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.378653 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.378694 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.382883 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.382885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.382886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.382894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.384539 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.389144 (+ 4605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.389145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.389145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.389209 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.389929 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.389930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.389930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.390071 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.390155 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.391238 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.391352 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:35:29.392357 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.368296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.24.11.188:38487)
0928 06:35:29.368331 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.368332 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.368980 (+ 648us) server_negotiation.cc:407] Connection header received
0928 06:35:29.370088 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.370089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.370102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.370132 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.371685 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.372421 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.377915 (+ 5494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.378217 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.378262 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.384945 (+ 6683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.384946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.384947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.384954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.386616 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.389425 (+ 2809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.389426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.389427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.389478 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.390349 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.390350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.390351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.390486 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.390558 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.392173 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.392256 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:35:29.393214 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:29.368426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.24.11.190:39903)
0928 06:35:29.368460 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.368462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.368890 (+ 428us) server_negotiation.cc:407] Connection header received
0928 06:35:29.369221 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.369222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.369232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.369262 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.374311 (+ 5049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.375046 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.376952 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:29.377243 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:29.377296 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:29.387037 (+ 9741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:35:29.387039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:35:29.387040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:35:29.387047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:35:29.388732 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.389690 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.389691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.389691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.389741 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:35:29.390723 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:35:29.390724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:35:29.390724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:35:29.390857 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:35:29.390955 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:29.393042 (+ 2087us) server_negotiation.cc:299] Negotiation successful
0928 06:35:29.393125 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:35:29.394884 15074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.394999 15074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:29.395009 15043 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.395097 15043 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:29.395162 15083 mvcc.cc:205] Tried to move back new op lower bound from 7334204741164482560 to 7334204741089337344. Current Snapshot: MvccSnapshot[applied={T|T < 7334204741164482560}]
I20260928 06:35:29.402180 15057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.402302 15057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557 [sys.catalog]: This master's current role is: LEADER
I20260928 06:35:29.402634 15029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "933e4d14c8c74898a7775f0e18335557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "933e4d14c8c74898a7775f0e18335557" member_type: VOTER last_known_addr { host: "127.24.11.190" port: 39903 } } peers { permanent_uuid: "a12a52cb456d42d0aaceefb34e420d85" member_type: VOTER last_known_addr { host: "127.24.11.189" port: 39139 } } peers { permanent_uuid: "1f0fbabaabea4f9f8ba203c8838ff187" member_type: VOTER last_known_addr { host: "127.24.11.188" port: 38487 } } }
I20260928 06:35:29.402732 15029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:35:29.403412 15081 mvcc.cc:205] Tried to move back new op lower bound from 7334204741164482560 to 7334204741089337344. Current Snapshot: MvccSnapshot[applied={T|T < 7334204741164482560}]
I20260928 06:35:29.403870 15076 catalog_manager.cc:1347] Generated new cluster ID: f9a2ea24497d4a8d92692d3290b86a5f
I20260928 06:35:29.403923 15076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:35:29.412070 15076 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:35:29.412827 15076 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:35:29.425428 15076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 933e4d14c8c74898a7775f0e18335557: Generated new TSK 0
I20260928 06:35:29.425665 15076 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:35:29.448715 14876 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.446979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.24.11.190:39903)
0928 06:35:29.447023 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.447024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.448053 (+ 1029us) server_negotiation.cc:407] Connection header received
0928 06:35:29.448158 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.448159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.448170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.448200 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.448645 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37210: BlockingRecv error: recv got EOF from 127.0.0.1:37210 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:35:29.449460 15027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.447407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.24.11.188:38487)
0928 06:35:29.447449 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.447452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.448953 (+ 1501us) server_negotiation.cc:407] Connection header received
0928 06:35:29.449048 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.449049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.449059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.449089 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.449400 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45140: BlockingRecv error: recv got EOF from 127.0.0.1:45140 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:35:29.450194 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.447734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.24.11.189:39139)
0928 06:35:29.447808 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.447810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.449672 (+ 1862us) server_negotiation.cc:407] Connection header received
0928 06:35:29.449798 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.449799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.449810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.449839 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.450135 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44718: BlockingRecv error: recv got EOF from 127.0.0.1:44718 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:35:29.452555 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.451821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.24.11.189:39139)
0928 06:35:29.451857 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.451859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.452033 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:35:29.452133 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.452135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.452156 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.452187 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.452511 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44724: BlockingRecv error: recv got EOF from 127.0.0.1:44724 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:29.455572 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.454822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.24.11.189:39139)
0928 06:35:29.454855 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.454857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.455070 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:35:29.455166 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.455167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.455177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.455205 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.455529 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44726: BlockingRecv error: recv got EOF from 127.0.0.1:44726 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:35:29.459357 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.458678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.24.11.189:39139)
0928 06:35:29.458710 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.458712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.458904 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:35:29.458986 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.458987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.458996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.459021 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.459304 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44736: BlockingRecv error: recv got EOF from 127.0.0.1:44736 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:35:29.465180 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.464452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.24.11.189:39139)
0928 06:35:29.464485 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.464487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.464704 (+ 217us) server_negotiation.cc:407] Connection header received
0928 06:35:29.464797 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.464798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.464807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.464836 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.465138 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44740: BlockingRecv error: recv got EOF from 127.0.0.1:44740 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:35:29.475682 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.474597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.24.11.189:39139)
0928 06:35:29.474659 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.474661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.474916 (+ 255us) server_negotiation.cc:407] Connection header received
0928 06:35:29.475034 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.475035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.475048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.475082 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.475627 (+ 545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44750: BlockingRecv error: recv got EOF from 127.0.0.1:44750 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:35:29.493690 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.492903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.24.11.189:39139)
0928 06:35:29.492950 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.492952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.493110 (+ 158us) server_negotiation.cc:407] Connection header received
0928 06:35:29.493186 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.493187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.493204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.493258 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.493621 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44764: BlockingRecv error: recv got EOF from 127.0.0.1:44764 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:35:29.527520 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.526743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.24.11.189:39139)
0928 06:35:29.526796 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.526798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.526941 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:35:29.527016 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.527017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.527029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.527094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.527446 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:35:29.593416 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.592680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.24.11.189:39139)
0928 06:35:29.592776 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.592778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.592810 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:35:29.592889 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.592890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.592900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.592964 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.593348 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44780: BlockingRecv error: recv got EOF from 127.0.0.1:44780 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 06:35:29.723394 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.722600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.24.11.189:39139)
0928 06:35:29.722697 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.722700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.722795 (+ 95us) server_negotiation.cc:407] Connection header received
0928 06:35:29.722900 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.722901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.722913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.722990 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.723326 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44796: BlockingRecv error: recv got EOF from 127.0.0.1:44796 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 06:35:29.981410 14953 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:35:29.980707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.24.11.189:39139)
0928 06:35:29.980788 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:29.980790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:29.980800 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:35:29.980854 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:29.980855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:29.980867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:29.980941 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:35:29.981333 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44800: BlockingRecv error: recv got EOF from 127.0.0.1:44800 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 06:35:30.306046 15040 catalog_manager.cc:1259] Loaded cluster ID: f9a2ea24497d4a8d92692d3290b86a5f
I20260928 06:35:30.306171 15040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: loading cluster ID for follower catalog manager: success
I20260928 06:35:30.306955 15040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: acquiring CA information for follower catalog manager: success
I20260928 06:35:30.307202 15040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f0fbabaabea4f9f8ba203c8838ff187: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:30.347781 15054 catalog_manager.cc:1259] Loaded cluster ID: f9a2ea24497d4a8d92692d3290b86a5f
I20260928 06:35:30.347868 15054 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: loading cluster ID for follower catalog manager: success
I20260928 06:35:30.348706 15054 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: acquiring CA information for follower catalog manager: success
I20260928 06:35:30.349059 15054 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a12a52cb456d42d0aaceefb34e420d85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:35:30.497730 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.494583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.24.11.189:39139)
0928 06:35:30.494723 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.494726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.494740 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:35:30.494791 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.494792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.494844 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.494964 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:30.495088 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.495668 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.496482 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.496731 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.496818 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.497090 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:30.497256 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:30.497354 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.497611 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.497635 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":130,"threads_started":1}
I20260928 06:35:30.506714 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.504158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.24.11.188:38487)
0928 06:35:30.504298 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.504300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.504401 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:35:30.504469 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.504470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.504508 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.504580 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:30.504666 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.505139 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.505947 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.506136 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.506189 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.506326 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:30.506458 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:30.506539 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.506599 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.506623 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260928 06:35:30.511193 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.508430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.24.11.189:39139)
0928 06:35:30.508524 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.508526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.508533 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:35:30.508560 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.508561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.508574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.508643 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:30.508800 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.509555 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.510374 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.510663 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.510722 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.510897 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:30.510930 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:30.510982 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.511090 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.511115 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 06:35:30.511270 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.508480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.24.11.188:38487)
0928 06:35:30.508522 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.508524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.508744 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:35:30.508806 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.508807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.508816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.508857 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:30.508959 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.509405 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.510444 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.510752 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.510808 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.510970 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:30.511004 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:30.511052 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.511142 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.511167 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:35:30.511741 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:35:30.508347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.24.11.190:39903)
0928 06:35:30.508547 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 06:35:30.508549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:35:30.508655 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:35:30.508722 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:35:30.508723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:35:30.508772 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:35:30.508846 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:35:30.509003 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.509757 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.510700 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:35:30.510969 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:35:30.511040 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:35:30.511312 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:35:30.511420 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:35:30.511484 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:35:30.511625 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:35:30.511652 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":53,"threads_started":1}
I20260928 06:35:30.513309 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14805
I20260928 06:35:30.518563 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14880
I20260928 06:35:30.523669 24622 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwumdS4/build/release/bin/kudu with pid 14955
2026-09-28T06:35:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2912 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5758 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (275171 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST